Demystifying chat & memory: stateless models, Context, and compression
How is a completely stateless next-token model wrapped into an assistant that supports multi-turn follow-ups and memory?
Now we enter system engineering outside the model. First, bust the biggest illusion: **an LLM has zero built-in conversation state or memory**. Every API call hits a completely stateless function that reads one input, emits one completion, and immediately forgets everything.
How does multi-turn chat work then? Pure illusion managed by the wrapper app: before sending your Nth question, the client silently concatenates turns 1 through N-1 into the Context payload and resends the entire transcript from scratch! When resending full history gets too expensive, the app summarizes older turns into a compact string called “Memory”.
Inference weights are frozen; "memory" is the client resending the entire messages[] array every turn
Every turn re-bills the entire conversation history; once truncated past the window limit, early turns vanish.
Model Weights Never Update After an API Call Completes
Send only the latest user message and the model has zero recollection of Turn 1.
Turn N Re-Transmits and Re-Bills Turns 1 Through N
Cumulative prompt tokens grow quadratically across long multi-turn sessions.
Pin System Rules, Compact Old Turns, and Persist Key Facts Externally
Long-term memory is an application-layer database pattern, not weight mutation.
First-principles breakdown: three steps
Recognize Statelessness: Every Call Starts Fresh
Model weights are frozen read-only during inference; without resending prior turns in the request, it remembers nothing.
Full Context Concatenation: Resending the Transcript
Prepending prior User/Assistant turns creates multi-turn continuity, at the cost of re-billing all earlier tokens on every new turn.
Memory Compression: Rolling Summaries & Fact Slots
When transcripts grow long, compact older turns into a short summary block injected at the top of the prompt.
Drag the conversation turn slider (Turn 1 → 12) and switch between [Stateless Raw], [Full History Resend], and [Summary Memory] to inspect the live HTTP payload and cumulative token bill.
[POST /v1/completions]
{"messages": [
{"role": "system", "content": "【压缩记忆摘要 Memory】用户正在评估 DeepSeek MLA 架构，已确认 128K 场景需求。"},
// 仅保留最近 2 轮原文 + 当前第 8 轮问题
{"role": "user", "content": "第 8 轮追问：那它的显存占用大吗？"}
]}At Turn 20, resending full history burns 50,400 cumulative tokens due to quadratic re-reading, whereas summary memory uses 6,800.
Prompt vs. Context
Both go into the same input buffer: instructions are called Prompt, while background materials and chat history are called Context.
Under the hood, the model receives both as a single contiguous sequence of Token IDs.LLM / Agent Memory
Not neural weight updates—just an external database or summary text pasted back into the prompt prefix on the next turn.
Once you see this, you know “memory poisoning” is fixed simply by editing or clearing that stored text.Prompt / Prefix Caching
Since the system prompt and prior turns stay identical across turns, servers reuse their precomputed KV Cache at a discount.
Always place static instructions first and dynamic inputs last to maximize prefix cache hits.Prompt is the instruction, Context is supplied background, and Memory is compressed history—to the stateless model, they are all just one flat string of input tokens.
When using a standard LLM API on Turn 10 of a conversation, why does the assistant still remember the name you told it on Turn 1?
Explanations, cases, and exercises are authored by Reallier. Links provide further learning. The workplace activity and people are fictional.