Training neural networks: loss, gradient descent, and parameter accounting
How do billions of random numbers step-by-step become weights that understand language?
A newly initialized model contains pure random noise. Training follows a 4-step loop: forward pass predicts output → Loss measures error → Backpropagation computes gradients → Gradient Descent updates weights by the learning rate.
Learning rate sets the step size downhill: too small crawls forever; too large overshoots the valley and oscillates or explodes. Architects also audit parameter bills by multiplying input and output dimensions of every matrix.
Forward pass computes Loss, backprop computes ∂L/∂w, and gradient descent updates weights
Weights are never hand-coded; they descend the loss surface step by step.
Quantify Prediction Error as a Differentiable Scalar
MSE for regression; Cross-Entropy (-log P) for next-token prediction.
Propagate Gradients Backward via Chain Rule
Computes the exact partial derivative for billions of weights simultaneously.
w_next = w - η·∇L · Balance Step Size & Parameter Footprint
Each layer holds ~12·d² weights plus gradients and Adam optimizer states.
First-principles breakdown: three steps
Measure Error with a Loss Function
The lower the probability assigned to the true next token, the higher the Cross-Entropy Loss.
Backpropagation & Learning Rate Control
Compute each weight’s gradient and update W = W - lr × grad; excessive learning rates cause divergence.
Audit Matrix Parameter Counts
A 768→3072 matrix holds 2.36M parameters; in a standard Transformer block, FFN holds twice as many parameters as Attention.
Drag the learning rate slider to inspect the 8-step gradient descent trajectory, then adjust d_model and FFN multiplier to audit parameter counts.
| Step | w | Gradient ∂L/∂w | Loss L(w) |
|---|---|---|---|
| #0 | -2 | -10 | 26 |
| #1 | 0.5 | -5 | 7.25 |
| #2 | 1.75 | -2.5 | 2.563 |
| #3 | 2.375 | -1.25 | 1.391 |
| #4 | 2.688 | -0.625 | 1.098 |
| #5 | 2.844 | -0.313 | 1.024 |
Raise learning rate η > 0.85 to observe oscillation; the right panel shows why 4× FFN accounts for 66.7% of per-block weights.
Gradient Descent & Explosion
Stepping downhill along derivatives is gradient descent; stepping too far and shooting error to infinity is explosion.
When training crashes (Loss = NaN), check learning rate and normalization first.Backpropagation
Using the chain rule backward from final error to assign blame to every upstream matrix weight.
Forward pass computes answers; backprop assigns blame and is disabled during inference.Parameter Count Bill
The sum of rows × columns across every weight matrix in the model.
A 7B model at FP16 (2 bytes/param) requires 14 GB of VRAM just to hold static weights.Training has no magic: define a differentiable loss function and step opposite the gradient with a stable learning rate.
In the gradient descent sandbox, what happens to the Loss when the learning rate is set too high (e.g., > 1.0)?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.