Learning to Learn

Vision Transformer (ViT): A Mathematical Explanation Vision Transformer (ViT) The Vision Transformer (ViT) is a deep learning model that applies the Transformer architecture—originally designed for language processing—to visual data. Unlike CNNs, which operate on local pixel neighborhoods, ViT divides an image into patches and models global relationships among them via self-attention. 1. Image to Patch Embeddings The input image: $$ \mathbf{x} \in \mathbb{R}^{H \times W \times C} $$ is divided into non-overlapping patches of size \( P \times P \), giving a total of $$ N = \frac{H \times W}{P^2} $$ patches. Each patch \( \mathbf{x}^{(i)} \) is flattened and linearly projected into a \( D \)-dimensional embedding: $$ \mathbf{e}^{(i)} = \mathbf{W}_{\text{embed}} \, \text{vec}(\mathbf{x}^{(i)}) \in \mathbb{R}^D, \quad i = 1, \dots, N $$ After stacking all patch embeddings, we form: $$ \mathbf{E} = [\mathbf{e}^{(1)}, \dots, \mathb...

Linear Regression: Mathematical Foundations Linear Regression: Mathematical Foundations Linear regression is a fundamental statistical technique used to predict a real-valued output \( y \in \mathbb{R} \) for a given input data point \( x \in \mathbb{R}^D \). It assumes that the expected value of the target variable is a linear function of the input features: $$ \mathbb{E}[y \mid x] = w^\top x $$ 1. Dataset Representation Let the training dataset be represented by a feature matrix: $$ X \in \mathbb{R}^{N \times D} $$ where \( N \) is the number of data points and \( D \) is the number of features. The dataset can be expressed as: $$ X = [x_1, x_2, \dots, x_D] $$ Each \( x_i \) (for \( i = 1, \dots, D \)) is a column vector representing one feature across all samples. 2. Model Formulation A general polynomial form of regression can be written as: $$ y = w_0 + w_{11} x_1 + w_{12} x_1^2 + \dots + w_{21} x_2 + w_{22}...

Basis Change and Matrix Approximation with SVD Basis Change and Matrix Approximation with SVD 1. Basis Change In this section, we’ll see how a transformation matrix changes when we change the basis of a linear mapping. Let there be a linear transformation: $$ \phi: V \rightarrow W $$ where \( V \in \mathbb{R}^n \) and \( W \in \mathbb{R}^m \). Let the ordered bases for \( V \) and \( W \) be: $$ B = (b_1, \dots, b_n), \quad C = (c_1, \dots, c_m) $$ and the new bases be: $$ \tilde{B} = (\tilde{b}_1, \dots, \tilde{b}_n), \quad \tilde{C} = (\tilde{c}_1, \dots, \tilde{c}_m) $$ If \( A \) represents the transformation matrix of \( \phi \) with respect to the bases \( (B, C) \), and \( \tilde{A} \) is the corresponding matrix with respect to \( (\tilde{B}, \tilde{C}) \), we aim to relate \( A \) and \( \tilde{A} \). 1.1 Relationship between old and new bases Each new basis vector in \( \tilde{B} \) can be written as a linear combi...

Sequence Networks Explained Sequence Networks Sequence networks can have either input as sequence, output as sequence, or both. We categorize them into three main types: Vec2Seq Seq2Vec Seq2Seq 1. Vec2Seq (Sequence Generation) $$ f_{\theta}:\mathbb{R}^{D}\to\mathbb{R}^{N_{\infty}\cdot C} $$ $$ p(y_{1:T}|x)=\sum_{h_{1:T}}p(y_{1:T},h_{1:T}|x)=\sum_{h_{1:T}}\prod_{t=1}^{T}p(y_{t}|h_{t})p(h_{t}|h_{t-1},y_{t-1},x) $$ Notation: \(h_t\): hidden state at time \(t\) \(p(h_1|h_0,y_0,x) = p(h_1|x)\): initial hidden state distribution For categorical and real-valued outputs: $$ p(y_t|h_t) = \text{Cat}(y_t | \text{softmax}(W_{hy} h_t + b_y)) $$ $$ p(y_t|h_t) = \mathcal{N}(y_t | W_{hy} h_t + b_y, \sigma^2 I) $$ This generative model is called a Recurrent Neural Network (RNN) . 2. Seq2Vec (Sequence Classification) $$ f_{\theta}:\mathbb{R}^{T D} \to \mathbb{R}^{C} $$ Output is a class label: \(y \in \{1, \dots, C\}\) ...

Dimensionality Reduction: Trade-offs and Techniques Dimensionality Reduction: Trade-offs and Techniques In this blog, we discuss some trade-offs involved when choosing a Dimensionality Reduction (DR) technique. Dimensionality reduction aims to represent high-dimensional data in a lower-dimensional space while preserving as much meaningful structure as possible. $$ f : \mathbb{R}^n \to \mathbb{R}^k, \quad k The goal is to find a mapping \( f \) that captures the essential relationships among the data points in \( \mathbb{R}^n \) within a lower-dimensional space \( \mathbb{R}^k \). However, depending on the technique used, distortions such as artificial clusters or misplaced neighbors may appear. We will focus on three widely used methods: t-SNE , UMAP , and TriMap . 1. t-SNE (t-distributed Stochastic Neighbor Embedding) t-SNE models the pairwise similarities between points in both the high-dimensional and low-dimensional spaces u...