Modern architecture evolution (Part 1): Sparse MoE and dynamic routing
Why do modern flagship models advertise “397B total parameters, 17B active”? How do they stay huge yet cheap to run?
In modern LLM evolution, engineers targeted the parameter-heavy FFN layer first. In a traditional Dense model, every single token activates 100% of parameters. Can a model hold 400B parameters of knowledge while activating only the relevant 17B per token?
Mixture of Experts (MoE) splits one giant FFN into many small FFN experts led by a lightweight trainable Router. A code token routes only to top-2 coding experts; a poetry token routes to literary experts. DeepSeekMoE refines this with finer experts plus an always-on Shared Expert for common grammar.
Replace dense FFN with N routed experts and activate only Top-K per token
MoE slashes per-token FLOPs while keeping total knowledge capacity high—at the cost of full VRAM residency.
Lightweight Linear Router Scores All N Experts
Dynamically dispatches each token based on its hidden representation.
Compute Only Top-K Experts; Remaining N-K Cost Zero FLOPs
Shared experts handle common syntax while routed experts specialize.
Total Params Dictate VRAM; Active Params Dictate FLOPs
GPU memory must hold all experts even though only a fraction fire per token.
First-principles breakdown: three steps
Split Giant FFN into Specialized Experts
Keep attention intact and split the heavy FFN into N smaller sub-networks that specialize during training.
Add a Trainable Gating Router
The router is a tiny linear layer + Softmax that scores all experts for the current token and selects only the Top-K.
Shared Expert + Fine-Grained Routing
Dedicate always-on shared experts to common language patterns so routed experts focus purely on specialized domains.
Switch between [Dense], [Standard MoE], and [DeepSeek Shared+Routed MoE] and click different token domains to inspect expert routing and active parameter bills.
Compare Dense, Standard MoE, and DeepSeekMoE: ~400B total capacity with only 17B–100B active parameters per token.
Dense vs. Sparse MoE Models
Dense activates 100% of weights for every token; Sparse MoE wakes up only a small fraction of expert weights per token.
Nearly all frontier large models today have transitioned to Sparse MoE architectures.Naming Convention “397B-A17B”
397B is total parameter count (VRAM footprint); A17B means only 17 billion parameters compute per token.
Size your GPU VRAM by the first number; judge inference speed and FLOP cost by the “A” number!Router Load Balancing
A training mechanism preventing the router from sending all tokens to the same two experts while others starve.
Keeping all experts balanced without routing bottlenecks is the core engineering craft of MoE.MoE trades VRAM capacity for compute speed: total parameters set knowledge capacity, while active parameters set inference speed and cost.
When an open-weights model is labeled “MoE Architecture: 236B Total, A21B Active”, what does that mean in practice?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.