OpenAI on NVIDIA Blackwell
Can I run gpt-oss 120B-A5.1B on an RTX 5060 Ti 16GB?
Not at Q4_K_M — it needs 67.4 GB against 14.2 GB available. You would need 5 of these cards.
Won't fit
67.4 GB of 14.2 GB · 473%070 GB
Weights
66.0 GB
KV cache
0.6 GB
Runtime overhead
0.8 GB
Over the limit
53.2 GB
Short by 53.2 GB. You can run it with 7 of 36 layers on the RTX 5060 Ti 16GB and the rest in system RAM, at roughly 15.5 tok/s — usable for batch work, painful for chat. A smaller quantisation or a shorter context is usually the better trade.
Generation15.5tok/s
Prompt processing1935tok/s
Max context0tokens
KV per 1K tokens0GB
Every quantisation of gpt-oss 120B-A5.1B on a RTX 5060 Ti 16GB
Highlighted row is the highest quality that still fits at 8K context.
| Quantisation | bpw | Weights | Total | Verdict | Max ctx | tok/s |
|---|---|---|---|---|---|---|
| FP16 / BF16 | 16.00 | 217.6 GB | 218.9 GB | Won't fit | — | 4.49 |
| INT8 / W8A8 | 8.50 | 115.3 GB | 116.7 GB | Won't fit | — | 8.63 |
| FP8 (E4M3) | 8.00 | 108.8 GB | 110.1 GB | Won't fit | — | 9.11 |
| Q6_K | 6.56 | 89.2 GB | 90.6 GB | Won't fit | — | 11.2 |
| Q5_K_M | 5.67 | 77.1 GB | 78.5 GB | Won't fit | — | 13.2 |
| Q5_K_S | 5.52 | 75.1 GB | 76.4 GB | Won't fit | — | 13.5 |
| Q4_K_M | 4.85 | 66.0 GB | 67.4 GB | Won't fit | — | 15.5 |
| Q4_K_S | 4.58 | 62.4 GB | 63.8 GB | Won't fit | — | 16.3 |
| Q4_0 | 4.55 | 62.0 GB | 63.4 GB | Won't fit | — | 16.4 |
| AWQ 4-bit | 4.25 | 59.4 GB | 60.7 GB | Won't fit | — | 17.1 |
| GPTQ 4-bit | 4.25 | 59.4 GB | 60.7 GB | Won't fit | — | 17.1 |
| MXFP4 | 4.25 | 59.4 GB | 60.7 GB | Won't fit | — | 17.1 |
| IQ4_XS | 4.25 | 58.0 GB | 59.3 GB | Won't fit | — | 17.4 |
| Q3_K_M | 3.91 | 53.4 GB | 54.7 GB | Won't fit | — | 19.2 |
| IQ3_M | 3.70 | 50.6 GB | 51.9 GB | Won't fit | — | 20.7 |
| IQ3_XXS | 3.06 | 41.9 GB | 43.3 GB | Won't fit | — | 25.7 |
| Q2_K | 2.63 | 36.1 GB | 37.5 GB | Won't fit | — | 30.0 |
| IQ2_XXS | 2.06 | 28.5 GB | 29.8 GB | Won't fit | — | 41.2 |
| IQ1_M | 1.75 | 24.3 GB | 25.7 GB | Won't fit | — | 51.7 |