Skip to content
TheQuantaSpace
← Research
AI & LLMs1 min read

The KV cache, explained from first principles

Why transformer inference gets faster after the first token, why long contexts eat GPU memory, and the back-of-envelope math you need to size a deployment.

Generating text with a transformer is autoregressive: each new token attends to every token before it. Done naively, producing token n means recomputing attention over all n − 1 previous tokens, again and again.

The trick

In self-attention, each token is projected into a query (Q), a key (K) and a value (V). The keys and values of past tokens never change once computed. So we compute them once and keep them around: that’s the KV cache.

At each decoding step we only compute Q, K and V for the new token, append its K and V to the cache, and attend over the cached history. Per-step work drops from quadratic to linear in sequence length.

The bill comes due in memory

The cache stores K and V for every layer, every head, every token:

bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × batch × bytes_per_value

For a model with 32 layers, 8 KV heads of dimension 128, and fp16 values (2 bytes), that’s 128 KiB per token. A single 32k-token conversation needs 4 GiB of cache, before you’ve batched a second user.

Where the field is going

  • Grouped-query attention (GQA) shares K/V across groups of query heads, shrinking the cache several-fold.
  • Paged attention allocates the cache in fixed blocks, like virtual memory, so fragmentation stops wasting GPU RAM.
  • Prefix caching reuses the cache for shared prompt prefixes, such as a long system prompt, across requests.
  • Quantized caches store K/V in 8-bit or lower with modest quality loss.

If you remember one thing: at long context, inference is a memory problem disguised as a compute problem.

  • #llm
  • #inference
  • #gpu