Sequence Networks Explained Sequence Networks Sequence networks can have either input as sequence, output as sequence, or both. We categorize them into three main types: Vec2Seq Seq2Vec Seq2Seq 1. Vec2Seq (Sequence Generation) $$ f_{\theta}:\mathbb{R}^{D}\to\mathbb{R}^{N_{\infty}\cdot C} $$ $$ p(y_{1:T}|x)=\sum_{h_{1:T}}p(y_{1:T},h_{1:T}|x)=\sum_{h_{1:T}}\prod_{t=1}^{T}p(y_{t}|h_{t})p(h_{t}|h_{t-1},y_{t-1},x) $$ Notation: \(h_t\): hidden state at time \(t\) \(p(h_1|h_0,y_0,x) = p(h_1|x)\): initial hidden state distribution For categorical and real-valued outputs: $$ p(y_t|h_t) = \text{Cat}(y_t | \text{softmax}(W_{hy} h_t + b_y)) $$ $$ p(y_t|h_t) = \mathcal{N}(y_t | W_{hy} h_t + b_y, \sigma^2 I) $$ This generative model is called a Recurrent Neural Network (RNN) . 2. Seq2Vec (Sequence Classification) $$ f_{\theta}:\mathbb{R}^{T D} \to \mathbb{R}^{C} $$ Output is a class label: \(y \in \{1, \dots, C\}\) ...
Math intensive