From Logits to the next token: Softmax, Temperature, and Top-P sampling
How do the 50,257 raw Logit scores turn into the actual next word on your screen?
After the final Linear projection, the network emits raw unconstrained scores called Logits. Softmax normalizes these scores into probabilities summing to 100%, and a decoding strategy samples the winning token.
Why does the same prompt yield different replies? Because decoding samples from probabilities rather than always taking the top token. Temperature scales logits before Softmax: low T sharpens the peak; high T flattens it. Top-K and Top-P (nucleus) truncate low-probability tail tokens to prevent gibberish.
Temperature reshapes candidate probabilities without changing model knowledge
Low temperature concentrates probability on the top token; high temperature flattens the curve.
Sharp Peak · Predictable Output
Best for factual extraction, formatting, and spreadsheet cleaning.
Balanced Fluency & Variety
Suited for drafting emails and rewriting notices while checking facts.
Flattened Curve · Higher Randomness
Increases variety for brainstorming but raises the risk of ungrounded wording.
First-principles breakdown: three steps
Convert Raw Logits to Probabilities via Softmax
Apply exp(z/T) so arbitrary real scores become positive probabilities summing to 1.
Reshape Concentration with Temperature
As T → 0, selection becomes deterministic; as T > 1, the distribution flattens for creative variety.
Prune Tail Noise with Top-K & Top-P
Zero out low-rank tail candidates and renormalize so random draws never pick nonsensical tokens.
Drag the Temperature and Top-K / Top-P sliders to watch candidate probabilities redistribute, then click “Draw 20 samples” to verify empirical hits.
Same scores, different probabilities.
| Candidate | Raw Softmax Prob | Renormalized Prob |
|---|---|---|
| coffee | 58.8% | 67.9% |
| milk | 27.8% | 32.1% |
| water | 11.6% | 0.0% |
| bread | 1.8% | 0.0% |
Candidates and scores are hand-set; probabilities and random draws are computed live. Top-K and Top-P zero out unlikely tail tokens.
Logits (Raw Unnormalized Scores)
The raw numbers straight out of the final linear layer (e.g., [2.4, 1.8, -0.4]) before percentage conversion.
If you only ever want the #1 token (greedy decoding), taking argmax of Logits directly is enough.Softmax Function
Exponentiate each score to make it positive, then divide by the total sum so probabilities add up to 100%.
Normalizes scores into a valid probability distribution while emphasizing top candidates.Top-P (Nucleus) Sampling
Sort candidates high-to-low and keep only the top subset whose cumulative probability reaches P (e.g., 90%).
Smarter than fixed Top-K: keeps 1–2 words when the model is confident and widens the pool when uncertain.Temperature and Top-P only change the lottery rules over candidates; lowering temperature increases consistency but never verifies facts.
Does lowering Temperature to 0.1 guarantee that the model’s answer is 100% factually accurate?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.