TrOCR: A Mathematical Explanation TrOCR TrOCR (Transformer-based Optical Character Recognition) maps an input image \( \mathbf{x} \) into a text sequence \( \mathbf{y} = (y_1, \dots, y_T) \). It models the conditional probability: $$ p(\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} p(y_t \mid y_{<t}, \mathbf{x}) $$ Notation: \( \mathbf{x} \): input image, \( y_t \): predicted token at step \( t \), \( y_{<t} \): sequence of previously generated tokens, \( T \): output sequence length. TrOCR consists of two major components: A Vision Transformer (ViT) as the encoder A Text Transformer as the decoder 1. Encoder — Vision Transformer (ViT) 1.1 Image to Patch Embeddings Input image: $$ \mathbf{x} \in \mathbb{R}^{H \times W \times 3} $$ The image is divided into \( N = \frac{H \times W}{P^2} \) patches of size \( P \times P \). Each patch is flattened and projected into a D-dimensional embedding: $$ \mathbf{z}_0^{...
Math intensive