FFN and Linear Projection: the true compute and parameter workhorse
Why is Attention just a context router while the Feed-Forward Network (FFN) does the heavy lifting?
Beginners often think Attention is the whole Transformer. In reality, Attention only gathers context; the Feed-Forward Network (FFN) that follows performs the deep non-linear feature transformation and holds ~2/3 of each block’s parameters.
Standard FFN uses an “expand-activate-contract” sandwich: project 768D up 4× to 3,072D, apply GELU to filter features in high-dimensional space, then project back to 768D. After N stacked blocks, a final Linear layer projects 768D to 50,257 vocabulary scores.
Attention gathers context; FFN expands 4× to fire factual memory slots before vocab projection
FFN holds ~2/3 of Transformer parameters and acts as a key-value memory bank.
Expand Context Vector into a 4× Wider Feature Space
Each hidden neuron acts as a pattern detector over the residual stream.
Sparse Activation Fires Matched Neurons and Zeroes Noise
Without non-linearity, W_down·W_up collapses into a low-rank linear matrix.
Project Back to Residual Stream and Score All Vocab Tokens
Unembedding maps the final hidden state to vocabulary probabilities.
First-principles breakdown: three steps
Up-Project (768 → 3072) to Expand Features
Expand the vector 4× so entangled semantic and syntactic features separate cleanly in a 3,072D latent space.
Activate and Down-Project (3072 → 768)
Filter through GELU and project back to 768D. Matching input/output dimensions lets Attention + FFN blocks stack like Lego bricks.
Final Linear Projection to Vocabulary
After all stacked blocks, multiply the final 768D vector by the projection matrix to produce 50,257 raw token scores (Logits).
Switch the expansion multiplier (1×, 4× 3072D, 8×) and toggle off GELU activation in the sandbox to see neuron filtering and matrix collapse warnings.
Uncheck GELU to verify how W1×W2 collapses into a single linear matrix without non-linear activation.
FFN / MLP (Feed-Forward Network)
A two-layer “expand → activate → contract” network applied independently to each token position.
Holds ~67% of block parameters and is the exact component replaced by Mixture-of-Experts (MoE).Transformer Block
One Multi-Head Attention layer + one FFN layer wrapped with residual connections and normalization.
Because input and output are both 768D, scaling depth simply means stacking this block dozens of times.Linear Projection / LM Head
The final activation-free matrix multiplication mapping hidden dimension (768) to vocabulary size (50,257).
Want sentiment classification instead? Just swap this head from 768→50,257 to 768→2.Attention mixes information horizontally across tokens; FFN refines features vertically within each token; Linear projects to the vocabulary.
Why does the FFN layer typically expand from 768 to 3,072 and then contract back to 768?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.