NVIDIA · Blackwell

What runs on a RTX 5090

32 GB at 1792 GB/s. The largest model that fits at Q4_K_M is Mixtral 8x7B, at roughly 120 tokens per second.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

6.4 GB of 29.3 GB · 22%

029 GB

Weights 4.6 GB

KV cache 1.0 GB

Runtime overhead 0.8 GB

Llama 3.1 8B at Q4_K_M leaves 22.8 GB spare on a RTX 5090. There is room to raise the context length or move up a quantisation level.

Generation222tok/s

Prompt processing5466tok/s

Max context128Ktokens

KV per 1K tokens0GB

Every quantisation of Llama 3.1 8B on a RTX 5090

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	15.0 GB	16.8 GB	Runs comfortably	108K	75.5
INT8 / W8A8	8.50	7.7 GB	9.5 GB	Runs comfortably	128K	140
Q8_0 (GGUF)	8.50	7.7 GB	9.5 GB	Runs comfortably	128K	140
FP8 (E4M3)	8.00	7.5 GB	9.3 GB	Runs comfortably	128K	144
Q6_K	6.56	6.1 GB	8.0 GB	Runs comfortably	128K	173
AWQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	128K	193
GPTQ 4-bit	4.25	5.4 GB	7.2 GB	Runs comfortably	128K	193
MXFP4	4.25	5.4 GB	7.2 GB	Runs comfortably	128K	193
Q5_K_M	5.67	5.3 GB	7.1 GB	Runs comfortably	128K	196
Q5_K_S	5.52	5.2 GB	7.0 GB	Runs comfortably	128K	201
Q4_K_M	4.85	4.6 GB	6.4 GB	Runs comfortably	128K	222
Q4_K_S	4.58	4.4 GB	6.2 GB	Runs comfortably	128K	231
Q4_0	4.55	4.4 GB	6.2 GB	Runs comfortably	128K	232
IQ4_XS	4.25	4.1 GB	5.9 GB	Runs comfortably	128K	244
Q3_K_M	3.91	3.8 GB	5.7 GB	Runs comfortably	128K	259
IQ3_M	3.70	3.7 GB	5.5 GB	Runs comfortably	128K	269
IQ3_XXS	3.06	3.2 GB	5.0 GB	Runs comfortably	128K	305
Q2_K	2.63	2.8 GB	4.6 GB	Runs comfortably	128K	335
IQ2_XXS	2.06	2.3 GB	4.2 GB	Runs comfortably	128K	385
IQ1_M	1.75	2.1 GB	3.9 GB	Runs comfortably	128K	420

Q4_K_M at 8K context. Bandwidth sets the speed; capacity sets the ceiling.

Model	Params	Weights	Verdict	Max ctx	tok/s
DeepSeek-R1 671B-A37B	671B	379.0 GB	Won't fit	—	2.10
Qwen3 235B-A22B	235.1B	132.8 GB	Won't fit	—	3.83
gpt-oss 120B-A5.1B	116.8B	66.0 GB	Won't fit	—	21.5
Llama 4 Scout 109B-A17B	109B	61.7 GB	Won't fit	—	6.75
Qwen2.5 72B	72.7B	41.2 GB	Won't fit	—	2.50
Llama 3.1 70B	70.5B	40.0 GB	Won't fit	—	2.65
Mixtral 8x7B	46.7B	26.4 GB	Fits, but tight	17K	120
Command R 35B	35.0B	20.1 GB	Won't fit	7K	12.7
Yi-1.5 34B	34.4B	19.5 GB	Runs comfortably	32K	56.8
Qwen3 32B	32.8B	18.6 GB	Runs comfortably	39K	59.0
Qwen2.5-Coder 32B	32.8B	18.6 GB	Runs comfortably	39K	59.0
DeepSeek-R1-Distill-Qwen 32B	32.8B	18.6 GB	Runs comfortably	39K	59.0
Qwen3 30B-A3B	30.5B	17.3 GB	Runs comfortably	120K	159
Gemma 3 27B	27.4B	15.6 GB	Runs comfortably	128K	70.7
Mistral Small 24B	23.6B	13.4 GB	Runs comfortably	32K	82.7
gpt-oss 20B-A3.6B	20.9B	11.9 GB	Runs comfortably	128K	311
Qwen3 14B	14.8B	8.5 GB	Runs comfortably	128K	126
Phi-4 14B	14.7B	8.4 GB	Runs comfortably	16K	125
Mistral NeMo 12B	12.3B	7.0 GB	Runs comfortably	128K	149
Gemma 3 12B	12.2B	7.0 GB	Runs comfortably	128K	151
GLM-4 9B	9.4B	5.4 GB	Runs comfortably	128K	203
Qwen3 8B	8.2B	4.7 GB	Runs comfortably	128K	214
Llama 3.1 8B	8.0B	4.6 GB	Runs comfortably	128K	222
DeepSeek-R1-Distill-Llama 8B	8.0B	4.6 GB	Runs comfortably	128K	222
Qwen2.5 7B	7.6B	4.4 GB	Runs comfortably	128K	247
Mistral 7B v0.3	7.3B	4.1 GB	Runs comfortably	32K	245
Gemma 3 4B	4.3B	2.5 GB	Runs comfortably	128K	405
Qwen3 4B	4.0B	2.3 GB	Runs comfortably	128K	380
Llama 3.2 3B	3.2B	1.8 GB	Runs comfortably	128K	478
Llama 3.2 1B	1.2B	0.7 GB	Runs comfortably	128K	1235