Pretraining & Few-Shot emergence: how next-token prediction builds a generalist
Without human labels, how does masking and predicting words across web text unlock translation and classification?
Once the Transformer pipeline is built, its training is remarkably simple: Pre-training needs zero human labels. Any raw sentence automatically yields N-1 training pairs by using each prefix to predict the next token—a paradigm called Self-Supervised Learning.
During inference we only use the last position, but during training every position predicts its next token in parallel. At scale, predicting the next token forces the network to internalize syntax, facts, and logic—unlocking Few-Shot In-Context Learning without updating a single weight.
Zero manual labels: raw text yields N-1 targets, unlocking in-context learning at scale
To minimize next-token loss across trillions of tokens, weights internalize world structure.
First k Tokens as Input X, Token k+1 as Ground-Truth Y
Eliminates human labeling bottlenecks across trillions of tokens.
Minimizing Cross-Entropy Compresses Factual & Logical Patterns
Predicting code or math tokens requires internalizing syntax and logic.
At Scale, Few-Shot Prompt Examples Switch Tasks Without Weight Updates
Induction heads copy and generalize task patterns directly from context.
First-principles breakdown: three steps
Self-Supervised Window: 1 Sentence = N-1 Targets
Raw text naturally provides (prefix → next token) targets; all positions compute loss in parallel during training.
Predicting Next Token = Compressing Knowledge
To score “Beijing” highest after “The capital of China is __”, weight matrices must compress world knowledge and grammar.
Few-Shot Learning: Zero Weight Updates
Supplying a few input→output examples in context lets attention lock onto the pattern and complete new tasks on the fly.
Edit the corpus sentence on the left to inspect all parallel self-supervised training pairs; switch 0/1/3-Shot on the right to see how frozen weights lock onto a task.
| Step | Context X | Target Y |
|---|---|---|
| #1 | 大 | → 模型 |
| #2 | 大 · 模型 | → 通 |
| #3 | 大 · 模型 · 通 | → 过 |
| #4 | 大 · 模型 · 通 · 过 | → 海 |
| #5 | 大 · 模型 · 通 · 过 · 海 | → 量 |
| #6 | 大 · 模型 · 通 · 过 · 海 · 量 | → 文 |
香蕉 -> banana 橘子 -> orange 葡萄 -> grape 苹果 ->
3-Shot: three examples lock both lowercase formatting and fruit semantics (97% apple).
Left: one raw sentence automatically generates multiple (context -> next token) training pairs; Right: 3-shot prompting locks translation probability to 97%.
Self-Supervised Learning
No human labels needed: hide the next word in raw text, let the model guess, and compare against the actual word.
Eliminating human labeling bottlenecks is what allowed models to train on trillions of web tokens.GPT (Generative Pre-trained Transformer)
Generative = token generation; Pre-trained = self-supervised reading on massive text; Transformer = the backbone architecture.
The three words in GPT literally spell out its task, training paradigm, and architecture.Few-Shot In-Context Learning
Zero parameter updates: just include 2–3 worked examples in the prompt and the model follows the pattern.
Ended the era of training separate models for every task, ushering in general-purpose foundation models.Self-supervised pretraining builds a generalist; Few-shot learning proves that next-token prediction can solve new tasks purely from context patterns.
Why is Transformer pre-training so efficient when processing a 100-token training passage?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.