How much VRAM does Llama 3.1 70B need?

At Q4_K_M the weights are 40.0 GB. With an 8K context the KV cache adds 2.5 GB and the runtime adds 0.6 GB, for 43.1 GB in total.

Llama on NVIDIA Blackwell

Can I run Llama 3.1 70B on an NVIDIA DGX Spark (GB10)?

Yes. Llama 3.1 70B at Q4_K_M uses 43.1 GB of the 96.0 GB available on a NVIDIA DGX Spark (GB10), and runs at about 4.37 tokens per second. You can push the context to 128K.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

43.1 GB of 96.0 GB · 45%

096 GB

Weights 40.0 GB

KV cache 2.5 GB

Runtime overhead 0.6 GB

Llama 3.1 70B at Q4_K_M leaves 52.9 GB spare on a NVIDIA DGX Spark (GB10). There is room to raise the context length or move up a quantisation level.

Generation4.37tok/s

Prompt processing372tok/s

Max context128Ktokens

KV per 1K tokens0GB

Every quantisation of Llama 3.1 70B on a NVIDIA DGX Spark (GB10)

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	131.4 GB	134.5 GB	Won't fit	—	0.69
INT8 / W8A8	8.50	69.3 GB	72.4 GB	Runs comfortably	83K	2.55
Q8_0 (GGUF)	8.50	69.3 GB	72.4 GB	Runs comfortably	83K	2.55
FP8 (E4M3)	8.00	65.7 GB	68.8 GB	Runs comfortably	95K	2.69
Q6_K	6.56	53.9 GB	57.0 GB	Runs comfortably	128K	3.27
Q5_K_M	5.67	46.6 GB	49.7 GB	Runs comfortably	128K	3.77
Q5_K_S	5.52	45.3 GB	48.4 GB	Runs comfortably	128K	3.87
Q4_K_M	4.85	40.0 GB	43.1 GB	Runs comfortably	128K	4.37
Q4_K_S	4.58	37.8 GB	40.9 GB	Runs comfortably	128K	4.61
AWQ 4-bit	4.25	37.8 GB	40.9 GB	Runs comfortably	128K	4.61
GPTQ 4-bit	4.25	37.8 GB	40.9 GB	Runs comfortably	128K	4.61
MXFP4	4.25	37.8 GB	40.9 GB	Runs comfortably	128K	4.61
Q4_0	4.55	37.6 GB	40.7 GB	Runs comfortably	128K	4.63
IQ4_XS	4.25	35.2 GB	38.3 GB	Runs comfortably	128K	4.94
Q3_K_M	3.91	32.5 GB	35.6 GB	Runs comfortably	128K	5.33
IQ3_M	3.70	30.8 GB	33.9 GB	Runs comfortably	128K	5.61
IQ3_XXS	3.06	25.7 GB	28.8 GB	Runs comfortably	128K	6.67
Q2_K	2.63	22.3 GB	25.4 GB	Runs comfortably	128K	7.63
IQ2_XXS	2.06	17.8 GB	20.9 GB	Runs comfortably	128K	9.45
IQ1_M	1.75	15.3 GB	18.4 GB	Runs comfortably	128K	10.9

Also worth checking

Llama 3.1 70B on every GPU
Full hardware table Everything that runs on a NVIDIA DGX Spark (GB10)
Full model table Why the KV cache is the part that bites
Guide