Compute, hardware, and quantization: GPU, SSD, and FP8/UE8M0
Why are CPUs so slow at LLMs, and what do you lose vs. gain when quantizing weights from FP32 to FP8 or UE8M0?
Understanding architecture requires grounding in physical hardware. 99% of neural network compute is embarrassingly parallel matrix multiplication. A CPU is a few PhDs solving equations serially; a GPU is 20,000 students multiplying numbers in lockstep.
The real bottleneck during token generation is Memory Bandwidth: every single token requires streaming all active weights from memory into compute cores. Streaming from SSD (7 GB/s) or DDR5 RAM (90 GB/s) starves the GPU! Quantizing weights from FP32 to BF16, FP8, or block-scaled formats (UE8M0) slashes memory size and multiplies token throughput.
Prefill is compute-bound, Decode is memory-bound; quantization shrinks VRAM and boosts tok/s
Generating each token streams all active weights from HBM—fewer bytes per weight means faster decoding.
Parallel Prompt Prefill Is GEMM; Token Decode Is GEMV
At small batch sizes, GPU ALUs sit idle waiting for HBM memory bandwidth.
Map 16-Bit Floats to 8/4-Bit Integer Grids via Group Scales
Group-wise scaling isolates outliers and preserves high signal-to-noise ratio.
70B Shrinks from 140GB to 35GB, Accelerating Decode Bandwidth ~4×
Enables 70B-class models to run on single workstation or consumer GPUs.
First-principles breakdown: three steps
GPU Parallelism vs. The Memory Bandwidth Wall
During decoding, weights must be read from memory for every token; HBM3e (>3,000 GB/s) outpaces SSD (7 GB/s) by orders of magnitude.
Anatomy of a Float: Sign, Exponent, Mantissa
Exponent bits set dynamic range while Mantissa bits set precision; neural weights care far more about range than fine decimal digits.
Quantizing from FP32 to FP8 & UE8M0
Trimming mantissa bits and using shared block scales (like UE8M0 8-bit unsigned exponents) halves memory with minimal accuracy loss.
Switch numeric precision (FP32 → BF16 → FP8 → UE8M0) and storage tiers (SSD → DDR5 → HBM3e) to inspect 70B model size, bit layout, and live Tokens/sec.
π ≈ 3.125 (|Error| = 0.016593)Switch to NVMe SSD to see why weights must live in GPU HBM: streaming 70B weights from disk caps decoding at <0.2 tokens/sec.
Tensor
0D is a scalar, 1D is a vector, 2D is a matrix, and 3D+ is called a Tensor—simply a multi-dimensional number array.
“Tensor Core” simply means hardware circuitry built specifically to multiply multi-dimensional arrays.Model Quantization (BF16 / FP8 / INT4)
Storing model weights with 16, 8, or 4 bits instead of 32 bits via controlled rounding.
Cuts a 70B model from 140 GB (FP16) to ~35 GB (4-bit), fitting on a single workstation.UE8M0 Block Scaling Format
Unsigned (0 sign) + 8 Exponent bits + 0 Mantissa bits: represents pure powers of 2 as a shared scale factor for a block of weights.
Turns expensive float scaling into fast bit-shifts for next-gen AI accelerators.LLM token decoding is memory-bandwidth bound: quantizing bit-width and using high-bandwidth memory directly dictate tokens per second.
Why does running a 70B model by streaming weights directly from an NVMe SSD generate less than one token per second?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.