Guide

How much VRAM do you need for a local LLM?

The short answer is about 0.6 GB per billion parameters at Q4 — and then the context length ruins it.

The rule of thumb people repeat is half a gigabyte per billion parameters at four-bit quantisation. It is close enough to be useful: an 8B model is around 4.9 GB, a 70B model around 42 GB. Where it falls apart is that nobody runs a model with no context.

Start with the weights

Weights are the predictable part. Multiply the parameter count by the bits per weight and divide by eight. A 32B model at Q4_K_M — 4.85 effective bits once you count the block scales — comes to roughly 19.8 GB. That number does not move once you have chosen a quantisation, and it is the same on every card.

Then add the part that grows

The KV cache holds one key and one value vector per token, per layer, for the whole conversation. It grows linearly with context length, and linearly again with the number of conversations you serve at once. On Llama 3.1 8B it costs 128 KB per token, which is a gigabyte at 8K context and sixteen gigabytes at 128K. On a model using full multi-head attention, such as Command R 35B, it is five times worse.

This is why people load a model successfully, chat happily for twenty minutes, and then hit an out-of-memory error. The weights fit. The conversation did not.

And the overhead nobody budgets for

Before a single weight loads, the CUDA context takes around 550 MB. The runtime then wants buffers of its own: llama.cpp is modest at a couple of hundred megabytes, while vLLM reserves well over a gigabyte for CUDA graphs and its scheduler. Activations during prefill add a few hundred megabytes more, scaled by the chunk size. If a monitor is plugged into the card, the desktop compositor and your browser are already holding several hundred megabytes before you start.

Assume you can use about 90% of the number printed on the box.

A worked example

Qwen3 32B at Q4_K_M on a 32 GB RTX 5090, with a 16K context:

  • Weights: 19.8 GB
  • KV cache at 16K: 4.0 GB
  • Runtime overhead: 0.9 GB
  • Total: 24.7 GB against roughly 29 GB usable

It fits, with room to raise the context to around 39K before it stops fitting. Take the same model to a 24 GB card and it still loads, but the context ceiling drops to a few thousand tokens — technically running, practically useless.

What to do when it does not fit

In rough order of what costs you least:

  • Quantise the KV cache to Q8_0. Halves the cache for no quality loss worth measuring. Do this first, always.
  • Shorten the context. Most work does not need 128K. Setting a realistic limit is free.
  • Step down one quantisation. Q5_K_M to Q4_K_M is a real but small quality change; anything below Q3 is not.
  • Choose a mixture-of-experts model. A 30B model with 3B active parameters needs the memory of a 30B model but runs at the speed of a 3B one, which is an excellent trade if you have capacity but not bandwidth.
  • Offload layers to system RAM. The last resort. System memory is roughly ten times slower than VRAM, so speed falls off a cliff.