Reading
Guides
How much VRAM do you need for a local LLM?
The short answer is about 0.6 GB per billion parameters at Q4 — and then the context length ruins it.The KV cache is the memory nobody budgets for
Weights are fixed. The KV cache grows with every token, and it is what actually causes out-of-memory errors.Which quantisation should you actually use?
Q4_K_M for almost everyone, Q5 or Q6 if you have room, AWQ if you are serving with vLLM. The reasoning, briefly.Choosing --max-model-len in vLLM without guessing
vLLM claims almost all your VRAM at startup. Here is how to work out the context length that will actually load.Why a 30B MoE model runs faster than an 8B dense one
Mixture-of-experts models need the memory of a large model and the bandwidth of a small one. That trade is a gift for some hardware and a trap for others.
The short answer is about 0.6 GB per billion parameters at Q4 — and then the context length ruins it.The KV cache is the memory nobody budgets for
Weights are fixed. The KV cache grows with every token, and it is what actually causes out-of-memory errors.Which quantisation should you actually use?
Q4_K_M for almost everyone, Q5 or Q6 if you have room, AWQ if you are serving with vLLM. The reasoning, briefly.Choosing --max-model-len in vLLM without guessing
vLLM claims almost all your VRAM at startup. Here is how to work out the context length that will actually load.Why a 30B MoE model runs faster than an 8B dense one
Mixture-of-experts models need the memory of a large model and the bandwidth of a small one. That trade is a gift for some hardware and a trap for others.