Vision Transformers

Vision transformer(ViT) is a transformer based deep learning model primarily used for image classification task. It processes images by dividing them into patches, then learns relationships between these patches using the Transformer architecture. After processing, it generates a classification output, just like other models designed for image classification, such as Convolutional Neural Networks (CNNs).

ViTs work in the following manner:

1. Patch embedding: It divides images into $P \times P$ patches. So, for a $H \times W \times C$ image, number of patches $N = \frac{ H \times W }{P^2}$ , then $Patch Embedding_i = Flatten(x_i) \cdot W_{embed} \in \mathbb{R}^D$ and after stacking these patch embeddings we will get patch embedding matrix $E \in \mathbb{R}^{N \times D}$ .

2. Positional embedding: Since Transformers don’t inherently handle spatial information like CNNs, positional encodings are added to each patch embedding to provide information about the position of each patch in the image. $Z = E + P \in \mathbb{R}^{N \times D}$ .

3. Attention mechanism:

$Q_h = ZW_{Q}^{h}$ ,

$K_h = ZW_{K}^{h}$ ,

$V_h = ZW_{V}^{h}$ ,

$Attention_h = softmax(\frac{Q_h \cdot K_{h}^{T}} {\sqrt{D_h}})$

$Output_h = Attention_h \cdot V_h \in \mathbb{R}^{N \times D_h}$ ,

$Multi Head Attention Output = Concat(Output_1, ..., Output_H)W_0 \in \mathbb{R}^{N \times D_h}$ .

4. Feed-forward network:

$FFN(Z) = max(0, ZW_1 + b_1)W_2 + b_2)$

5. Final logits:

$logits = z_{CLS}W_{cls} \in \mathbb{R}^C$

6. Loss function:

$L = - \sum_{i=1}^{C}y_ilog(softmax(logits)_i)$

Sources:

1. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Comments

Wow concepts!!

Basis Change and Matrix Approximation with SVD

Basis Change: In this blog we will see how the transformation matrix for any linear mapping changes with the change in the basis. Let's assume we have a linear mapping with & , ordered bases for vector spaces and are and respectively and we are changing these bases to and for vector spaces and respectively. Also assume the transformation matrix in case of bases & is and after changing the bases to & it becomes . In this blog I will be taking subscripts , , and to represent vectors in the bases , , and respectively. As we know basis vectors spans the entire vector space, so will span the entire vector space , so will also span the basis vectors of . So we can say that each new basis vector of will be the linear combinations of the basis vectors of and we can write it as, . We store these coefficients in column-wise manner in a matrix (say ), so entries in the column of will be the coef...

Explore more 🐱‍🏍

Dimensionality Reduction

Hi, In this blog I will be discussing some trade-offs we make while choosing a dimensionality reduction technique for our problem. Now, let's jump into this directly. Dimensionality reduction(DR) reduces higher dimensional data to lower dimensions. Or we can say that DR maps -dimensional data into -dimensions ( ), ( ), where these new -dimensions hold nearly all of the relevant information about the original data. Sometimes DR results can show clusters of data that are not even present in the original data and sometimes it can map two neighbors from the higher dimension far into the lower dimension. So let's discuss and compare some methods which can prevent these problems. I will be discussing t-SNE, UMAP, and TriMap in this blog. 1. t-SNE (t-distributed Stochastic Neighborhood Embedding) t-SNE uses the distance between two points in higher dimensions and maps it to the lower dimension. where is computed by using binary search in the equation, Perplexity = ...

Explore more 🐱‍🏍

Linear Regression

Linear regression is used to predict real-valued output for a given input data point . Linear regression establishes a relationship of dependent variable with the features of the input data with an assumption that the expected value of the output(dependent variable) is a linear function of the input ( ). Let's assume our training dataset is where is the number of data points and is the number of dimension or number of features in our dataset. From now on we will write our dataset as where each for is a column vector. We can write the output as: or we can write it as: Before computing the final weights for this equation, we need to figure out what degree we should choose. We usually select the degree for which we get less mean squared error(MSE). The most common form of linear regression is degree 1 form: There are two ways by which we can estimate the parameters: Normal equation: Weight vector is estimated by matrix multiplication o...

Explore more 🐱‍🏍

Sequence Networks

Hi, in this blog I will be covering most of the sequence networks. Sequence networks can have either input as sequence or output as sequence or both(input and output) as sequences. We can sub-divide these sequence networks in the following three ways: Vec2Seq Seq2Vec Seq2Seq 1. Vec2Seq(Sequence Generation): where is hidden state and (initial hidden state distribution). For categorical and real-valued output, the distributions are given by: The above generative model is called Recurrent Neural Network(RNN) . 2. Seq2Vec(Sequence Classification): In classification task, the output is class label, We get better results if we let the hidden states depends on past as well as future context bidirectional RNN . Then we define, hidden state at time , . where 3. Seq2Seq(Sequence Translation): Aligned case where the initial state, ...

Explore more 🐱‍🏍

Global and Local Models

Recently we have seen a number of chatbots(chatGPT, BARD etc). These comes under LLMs(large language models) which are trained on huge datasets. I will discuss the methods which these models follow to give personalized and accurate responses. I will break this into three steps: Information gathering: gathers users search history, topics discussed, queries etc Information clustring: clusters related concepts, facts and ideas Personalized responses: searches information cluster to find most relevant and helpful information The information clustering process is a knowledge acquisition and adaptation process, which shares some similarity with online learning but has some key differences. Knowledge acquisition(KA) vs subset selection(SS): KA is a broader concept, it encompasses various methods and processes of gathering and accumulating new knowledge or information. This includes learning from experiences, interactions, readings, and various sources while SS is a narrower concepts. It re...

Explore more 🐱‍🏍

Learning to Learn

Search This Blog