Guide

The KV cache is the memory nobody budgets for

Weights are fixed. The KV cache grows with every token, and it is what actually causes out-of-memory errors.

Attention needs to compare the token being generated against every token before it. Rather than recompute those key and value projections each step, the runtime keeps them. That store is the KV cache, and its size is entirely predictable:

bytes = 2 × layers × context × kv_heads × head_dim × bytes_per_element × sequences

The leading 2 is for keys and values. Everything else comes from the model's configuration file.

Grouped-query attention changed the picture

Older models gave every attention head its own key and value. Llama 2 70B had 64 heads, and its cache was enormous. Grouped-query attention lets several query heads share one key-value head, and Llama 3.1 70B dropped to eight — an eightfold reduction, for essentially no quality cost. This is why a modern 70B model is more practical at long context than a 34B model from two years earlier.

When you look at a model's config, num_key_value_heads matters more for your memory budget than the parameter count does.

Latent attention goes further

DeepSeek's multi-head latent attention compresses keys and values into a single low-rank vector per layer, stored at 576 elements rather than a full set of head-sized tensors. The result is that DeepSeek-R1, at 671B parameters, has a KV cache comparable to a 70B model's. The weights are still a problem. The cache is not.

Sliding windows cap the growth

Gemma 3 makes most of its layers attend only to the last 1024 tokens, with every sixth layer seeing everything. At 8K context this barely helps. At 128K it means the cache grows by a factor of about six rather than sixteen. If you plan to work at very long context, this architectural detail matters more than any quantisation choice.

Quantising the cache

The cache can be stored at lower precision, exactly like weights. Q8_0 halves it and the quality difference is not measurable in normal use — there is no good reason not to enable it. Q4_0 quarters it and does cost some recall accuracy on long documents, which is a fair trade when the alternative is not running at all.

# llama.cpp
llama-server -m model.gguf --cache-type-k q8_0 --cache-type-v q8_0

# vLLM
vllm serve model --kv-cache-dtype fp8

Batching multiplies everything

Serving eight users at once means eight caches. On a 70B model at 8K context that is twenty gigabytes of cache alone. Paged attention in vLLM reduces the waste from padding but does not change the fundamentals: concurrency is paid for in memory, and it is the first thing to check when a server that worked in testing falls over in production.