How much VRAM does Qwen2.5 7B need?

At Q4_K_M the weights are 4.4 GB. With an 8K context the KV cache adds 0.4 GB and the runtime adds 0.8 GB, for 5.6 GB in total.

Qwen on NVIDIA Ada

Can I run Qwen2.5 7B on an L40S?

Yes. Qwen2.5 7B at Q4_K_M uses 5.6 GB of the 44.3 GB available on a L40S, and runs at about 110 tokens per second. You can push the context to 128K.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

5.6 GB of 44.3 GB · 13%

044 GB

Weights 4.4 GB

KV cache 0.4 GB

Runtime overhead 0.8 GB

Qwen2.5 7B at Q4_K_M leaves 38.7 GB spare on a L40S. There is room to raise the context length or move up a quantisation level.

Generation110tok/s

Prompt processing4988tok/s

Max context128Ktokens

KV per 1K tokens0GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	14.2 GB	15.4 GB	Runs comfortably	128K	35.7
INT8 / W8A8	8.50	7.3 GB	8.5 GB	Runs comfortably	128K	68.2
Q8_0 (GGUF)	8.50	7.3 GB	8.5 GB	Runs comfortably	128K	68.2
FP8 (E4M3)	8.00	7.1 GB	8.3 GB	Runs comfortably	128K	70.0
Q6_K	6.56	5.8 GB	7.1 GB	Runs comfortably	128K	84.6
AWQ 4-bit	4.25	5.3 GB	6.5 GB	Runs comfortably	128K	93.1
GPTQ 4-bit	4.25	5.3 GB	6.5 GB	Runs comfortably	128K	93.1
MXFP4	4.25	5.3 GB	6.5 GB	Runs comfortably	128K	93.1
Q5_K_M	5.67	5.0 GB	6.3 GB	Runs comfortably	128K	97.1
Q5_K_S	5.52	4.9 GB	6.1 GB	Runs comfortably	128K	99.6
Q4_K_M	4.85	4.4 GB	5.6 GB	Runs comfortably	128K	110
Q4_K_S	4.58	4.2 GB	5.4 GB	Runs comfortably	128K	116
Q4_0	4.55	4.2 GB	5.4 GB	Runs comfortably	128K	116
IQ4_XS	4.25	3.9 GB	5.2 GB	Runs comfortably	128K	122
Q3_K_M	3.91	3.7 GB	4.9 GB	Runs comfortably	128K	130
IQ3_M	3.70	3.5 GB	4.8 GB	Runs comfortably	128K	136
IQ3_XXS	3.06	3.0 GB	4.3 GB	Runs comfortably	128K	156
Q2_K	2.63	2.7 GB	3.9 GB	Runs comfortably	128K	173
IQ2_XXS	2.06	2.3 GB	3.5 GB	Runs comfortably	128K	202
IQ1_M	1.75	2.0 GB	3.3 GB	Runs comfortably	128K	222