How much VRAM does Qwen3 32B need?

At Q4_K_M the weights are 18.6 GB. With an 8K context the KV cache adds 2.0 GB and the runtime adds 0.8 GB, for 21.5 GB in total.

Qwen on NVIDIA Ada

Can I run Qwen3 32B on an L40S?

Yes. Qwen3 32B at Q4_K_M uses 21.5 GB of the 44.3 GB available on a L40S, and runs at about 26.2 tokens per second. You can push the context to 99K.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

21.5 GB of 44.3 GB · 48%

044 GB

Weights 18.6 GB

KV cache 2.0 GB

Runtime overhead 0.8 GB

Qwen3 32B at Q4_K_M leaves 22.9 GB spare on a L40S. There is room to raise the context length or move up a quantisation level.

Generation26.2tok/s

Prompt processing1160tok/s

Max context99Ktokens

KV per 1K tokens0GB

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	61.0 GB	63.9 GB	Won't fit	—	1.63
INT8 / W8A8	8.50	32.1 GB	34.9 GB	Runs comfortably	46K	15.6
Q8_0 (GGUF)	8.50	32.1 GB	34.9 GB	Runs comfortably	46K	15.6
FP8 (E4M3)	8.00	30.5 GB	33.3 GB	Runs comfortably	52K	16.3
Q6_K	6.56	25.0 GB	27.9 GB	Runs comfortably	74K	19.8
Q5_K_M	5.67	21.6 GB	24.5 GB	Runs comfortably	87K	22.7
Q5_K_S	5.52	21.1 GB	23.9 GB	Runs comfortably	90K	23.3
Q4_K_M	4.85	18.6 GB	21.5 GB	Runs comfortably	99K	26.2
AWQ 4-bit	4.25	18.3 GB	21.2 GB	Runs comfortably	101K	26.5
GPTQ 4-bit	4.25	18.3 GB	21.2 GB	Runs comfortably	101K	26.5
MXFP4	4.25	18.3 GB	21.2 GB	Runs comfortably	101K	26.5
Q4_K_S	4.58	17.6 GB	20.5 GB	Runs comfortably	103K	27.5
Q4_0	4.55	17.5 GB	20.4 GB	Runs comfortably	104K	27.7
IQ4_XS	4.25	16.4 GB	19.3 GB	Runs comfortably	108K	29.4
Q3_K_M	3.91	15.2 GB	18.0 GB	Runs comfortably	113K	31.6
IQ3_M	3.70	14.4 GB	17.3 GB	Runs comfortably	116K	33.1
IQ3_XXS	3.06	12.1 GB	15.0 GB	Runs comfortably	125K	38.9
Q2_K	2.63	10.6 GB	13.4 GB	Runs comfortably	128K	44.1
IQ2_XXS	2.06	8.5 GB	11.3 GB	Runs comfortably	128K	53.5
IQ1_M	1.75	7.4 GB	10.2 GB	Runs comfortably	128K	60.6