How text becomes numbers: tokenization and high-dimensional embeddings
Why can’t a single ID number represent a word, and why map each token into a 768D vector?
Neural networks only compute numbers. Feeding text takes two steps: Tokenization splits text into token IDs; then an Embedding Matrix maps each ID into a high-dimensional float vector (e.g., 768D in GPT-2).
A single number on a 1D line cannot separate multi-attribute concepts; One-Hot vectors explode to 50,000 dimensions with zero dot product between any two words. Dense trainable embeddings place semantically related words close together in vector space.
Models read numbered text chunks (tokens) rather than raw characters
Different splitting rules change chunk counts and boundaries, affecting context length and cost.
Continuous String: Chinese, English, Punctuation & Emoji
Humans read continuous sentences; models require discrete token sequences.
Fine Granularity · Longer Sequence
Every character becomes a separate unit, lengthening the sequence.
Frequent Phrases Grouped · Compact Sequence
Common phrases stay together as single tokens, balancing efficiency and coverage.
First-principles breakdown: three steps
Tokenization Sets the Atomic Unit
Subword tokenizers (like BPE) split text into frequent chunks. Token count differs from character count and drives billing.
From 1D Inseparable to High-Dim Separable
Interleaved points on a 1D line cannot be split by a line; lifting to higher dimensions makes features linearly separable.
End-to-End Trainable Embedding Matrix
Instead of hand-crafting features, nn.Embedding(Vocab, Dim) starts random and learns semantic directions end-to-end.
Edit the sentence and switch splitting rules first; then drag the Food and Tech sliders for “Apple” to watch its cosine similarity with Banana, Xiaomi, and GPU update live.
| Reference Word | 2D Vector [Fruit, Tech] | Cosine Similarity with “Apple” |
|---|---|---|
| Banana (Fruit) | [0.95, 0.05] | 0.997 |
| Xiaomi (Tech) | [0.08, 0.94] | 0.213 |
| GPU (Hardware) | [0.02, 0.98] | 0.15 |
Illustrative tokenizer · ␠ represents a space. Below, live cosine similarity shows how shifting dimensions moves “Apple” toward fruit or tech.
One-Hot Encoding
A 50,257-length array of zeros with a single 1 at the word’s index.
Wastes dimensions and yields 0 dot product between all word pairs.Embedding Matrix (nn.Embedding)
A lookup table of shape [VocabSize × HiddenDim]; Token ID N simply fetches row N.
Essentially a bias-free linear layer whose rows update during training.Cosine Similarity / Dot Product
Measures directional alignment between two vectors: near 1 means similar meaning, near 0 means unrelated.
Both Attention scoring and RAG vector search rely on this calculation.A Token ID is merely a row index; what actually enters the network is the high-dimensional embedding vector fetched from that row.
Will every model tokenize the same sentence into the same token count and embedding dimension?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.