Practised 0 / 16Next →
09 / 16 · MODULE 03 · SFT, COT & RLVR POST-TRAINING

Post-training trio: SFT, Chain-of-Thought (CoT), and RLVR

Why does a raw pre-trained model keep writing more exam questions instead of answering, and how does it learn to reason?

TAKE AWAYSwitch across Base, SFT, CoT, and RLVR stages on a business math problem to inspect output streams and accuracy

A raw pre-trained Base Model is a pure text completer: ask it a math question and it often continues writing Question 2 and Question 3! Turning this completer into a helpful reasoning assistant requires Post-training.

Post-training unfolds in three stages: 1) Supervised Fine-Tuning (SFT) teaches Q&A turn-taking; 2) Chain-of-Thought (CoT) externalizes intermediate calculation steps as context tokens; 3) RLHF / RLVR (verifiable rewards) reinforces self-verification and backtracking.

DIAGRAM · FOUR-STAGE POST-TRAINING PIPELINE

Pretraining builds knowledge, SFT teaches dialogue format, DPO aligns preferences, RLVR unlocks reasoning

Each post-training stage solves a distinct failure mode of the raw base model.

STAGE 1 · SUPERVISED FINE-TUNINGTurn a Document Completer into a Helpful Assistant via DemosActivates pre-existing knowledge and enforces instruction-following format.→STAGE 2 · PREFERENCE ALIGNMENTContrast Chosen vs Rejected Outputs with KL RegularizationShapes safety and tone while KL penalty prevents model degeneration.→STAGE 3 · VERIFIABLE REWARD RLReward Verified Math/Code Outcomes to Elicit Self-CorrectionModels autonomously discover verification and backtracking strategies.
STAGE 1 · SUPERVISED FINE-TUNING

Turn a Document Completer into a Helpful Assistant via Demos

Activates pre-existing knowledge and enforces instruction-following format.

STAGE 2 · PREFERENCE ALIGNMENT

Contrast Chosen vs Rejected Outputs with KL Regularization

Shapes safety and tone while KL penalty prevents model degeneration.

STAGE 3 · VERIFIABLE REWARD RL

Reward Verified Math/Code Outcomes to Elicit Self-Correction

Models autonomously discover verification and backtracking strategies.

First-principles breakdown: three steps

  1. SFT: Shift from Continuation to Q&A Alignment

    Pre-training uses massive raw text; SFT uses curated instruction-response pairs so the model answers questions directly.

  2. CoT: Use Context as an External Scratchpad

    A single forward pass has fixed depth; writing out step-by-step tokens lets subsequent tokens attend to intermediate results.

  3. RLHF & RLVR: Reinforce Self-Correction

    Rewarding verifiable correct answers (RLVR) lets the model autonomously discover verification and backtracking strategies.

Switch between [Base Pretrain], [SFT Direct], [CoT Scratchpad], and [RLVR Verified] in the sandbox to see how each handles the CNY 1,500 server cost trap.

TRY IT YOURSELF · LIVE CODE SANDBOXLIVE CALCULATION
Raw Web Next-Token ContinuationAcc 8%

“...What is the net profit? Question 2: An e-commerce site has 5,000 DAU...” (Continues writing exam questions!)

✕ Continuation or Calculation Flaw
Supervised Fine-Tuning (SFT Q&A Pairs)Acc 45%

“The monthly net profit is CNY 4,800.” (Answers directly without scratchpad tokens, forgetting the CNY 1,500 cost.)

✕ Continuation or Calculation Flaw
Chain-of-Thought (CoT External Scratchpad)Acc 84%

“1. Paying users = 1200×5% = 60; 2. Revenue = 60×80 = 4800; 3. Net = 4800 - 1500 = CNY 3,300.”

✓ Correct Net Profit CNY 3,300
Reinforcement Learning with Verifiable Rewards (RLVR)Acc 96%

“<think>Check unit scope: 60×80 = 4800 revenue; subtract 1500 server cost = 3300. Verify: 3300+1500=4800.</think> CNY 3,300.”

✓ Correct Net Profit CNY 3,300

Selected stage accuracy: 96%。Multi-step math requires emitting intermediate steps as context tokens (CoT / RLVR) so subsequent tokens can attend to them.

JARGON BUSTER · STRIP AWAY THE HYPE

Supervised Fine-Tuning (SFT)

Training a pre-trained model on tens of thousands of high-quality “Prompt + Ideal Response” pairs.

Same architecture, curated Q&A data—turning a text completer into a chat assistant.

Chain of Thought (CoT)

Having the model emit step-by-step reasoning tokens before stating the final conclusion.

Since each token gets one forward pass, emitting steps trades extra tokens for deeper compute.

RLHF & RLVR

RLHF scores responses via human preference; RLVR rewards objectively verifiable math or code outcomes.

Frontier model gaps today are increasingly driven by post-training RL quality.
ONE IDEA TO KEEP

LLMs cannot “silently do mental math” across steps—unless intermediate steps are emitted as tokens into context, they are forced to guess in a single forward pass.

CHECK YOUR UNDERSTANDING

Why does having an LLM write out intermediate reasoning steps (CoT) before the final answer dramatically boost accuracy on complex math problems?

Why does having an LLM write out intermediate reasoning steps (CoT) before the final answer dramatically boost accuracy on complex math problems?
REFERENCES & FURTHER LEARNING
Chain-of-Thought Prompting Elicits Reasoning in LLMs ↗

Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.