Everything is a function: from linear fit to giant classification
Why is a large language model fundamentally a function that classifies the next token?
The key to AI principles is defining inputs and outputs first. An LLM solves a simple problem: given context X, predict the next token Y. A neural network is a universal function template filled with unknown weights W and biases B.
Pure linear layers (W·X + B) always collapse into a straight line. Adding non-linear activations (ReLU/GELU) bends lines to fit complex curves. At the end, changing the output layer size adapts the same network from 2-way sentiment to 50,257-way next-token prediction.
A linear layer draws a line; stacking ReLU hinges approximates any continuous curve
Weights control slope, biases shift hinge positions, and ReLU breaks linear collapse.
Weighted Sum & Shift Only — Stacking Collapses to One Line
Without non-linearity, W₂(W₁x) = W_eff·x regardless of depth.
Zero Out Negative Region to Create a Local Hinge
Each hidden unit contributes a tunable hinge on the input axis.
Combine Hinges to Fit Complex Non-Linear Functions
Billions of parameters sculpt a high-dimensional piecewise manifold.
First-principles breakdown: three steps
Define Input X and Output Y First
Frame text generation as a clear interface: input is numbered context; output is a vector of 50,257 scores where the highest score points to the next token.
Break Linearity with Activations
Multiplying matrices without activations collapses to a single linear layer; ReLU or GELU lets the network bend and fit complex patterns.
Projection Head Defines the Task
The backbone extracts features; projecting to 2 numbers yields sentiment classification, while projecting to 50,257 yields next-token generation.
Drag weights W1, W2, and bias B in the code sandbox, toggle Linear vs Non-Linear activation to watch MSE loss, and switch output classification heads.
| x | Target y* | Network Output ŷ | Error |y* - ŷ| |
|---|---|---|---|
-2 | 3 | 2 | 1 |
-1 | 0.8 | 1.2 | 0.4 |
0 | -0.2 | 0.4 | 0.6 |
1 | 0 | 1 | 1 |
2 | 1.4 | 2.2 | 0.8 |
Toggle between Linear and ReLU/GELU to verify how non-linear hinges approximate curved targets.
Model Weights & Parameters
Simply the unknown coefficients W and B in Y = W·X + B, initialized randomly and tuned by data.
“70B parameters” simply means 70 billion floating-point knobs in the formula.Activation Function (ReLU / GELU)
A “bender” applied after linear sums that suppresses negative noise and preserves positive signals.
Without it, a 100-layer deep network collapses into a 1-layer linear equation.Autoregressive Generation
Pick 1 token from the function output, append it to the input, and run the exact same function again.
Streaming chat output is just a loop calling this classification function token by token.Strip away the scale: an LLM is still a function using non-linear activations to output scores across a vocabulary-sized vector.
What happens if you remove all non-linear activation functions from a 12-layer neural network?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.