OpenAI on NVIDIA Ampere
Can I run gpt-oss 120B-A5.1B on an RTX 3060 12GB?
Not at Q4_K_M — it needs 67.4 GB against 10.5 GB available. You would need 7 of these cards.
Won't fit
67.4 GB of 10.5 GB · 643%070 GB
Weights
66.0 GB
KV cache
0.6 GB
Runtime overhead
0.8 GB
Over the limit
56.9 GB
Short by 56.9 GB. You can run it with 4 of 36 layers on the RTX 3060 12GB and the rest in system RAM, at roughly 13.6 tok/s — usable for batch work, painful for chat. A smaller quantisation or a shorter context is usually the better trade.
Generation13.6tok/s
Prompt processing1029tok/s
Max context0tokens
KV per 1K tokens0GB
Every quantisation of gpt-oss 120B-A5.1B on a RTX 3060 12GB
Highlighted row is the highest quality that still fits at 8K context.
| Quantisation | bpw | Weights | Total | Verdict | Max ctx | tok/s |
|---|---|---|---|---|---|---|
| FP16 / BF16 | 16.00 | 217.6 GB | 218.9 GB | Won't fit | — | 4.15 |
| INT8 / W8A8 | 8.50 | 115.3 GB | 116.7 GB | Won't fit | — | 7.77 |
| FP8 (E4M3) | 8.00 | 108.8 GB | 110.1 GB | Won't fit | — | 8.39 |
| Q6_K | 6.56 | 89.2 GB | 90.6 GB | Won't fit | — | 10.1 |
| Q5_K_M | 5.67 | 77.1 GB | 78.5 GB | Won't fit | — | 11.8 |
| Q5_K_S | 5.52 | 75.1 GB | 76.4 GB | Won't fit | — | 12.1 |
| Q4_K_M | 4.85 | 66.0 GB | 67.4 GB | Won't fit | — | 13.6 |
| Q4_K_S | 4.58 | 62.4 GB | 63.8 GB | Won't fit | — | 14.6 |
| Q4_0 | 4.55 | 62.0 GB | 63.4 GB | Won't fit | — | 14.7 |
| AWQ 4-bit | 4.25 | 59.4 GB | 60.7 GB | Won't fit | — | 15.3 |
| GPTQ 4-bit | 4.25 | 59.4 GB | 60.7 GB | Won't fit | — | 15.3 |
| MXFP4 | 4.25 | 59.4 GB | 60.7 GB | Won't fit | — | 15.3 |
| IQ4_XS | 4.25 | 58.0 GB | 59.3 GB | Won't fit | — | 15.6 |
| Q3_K_M | 3.91 | 53.4 GB | 54.7 GB | Won't fit | — | 17.2 |
| IQ3_M | 3.70 | 50.6 GB | 51.9 GB | Won't fit | — | 18.0 |
| IQ3_XXS | 3.06 | 41.9 GB | 43.3 GB | Won't fit | — | 21.6 |
| Q2_K | 2.63 | 36.1 GB | 37.5 GB | Won't fit | — | 25.7 |
| IQ2_XXS | 2.06 | 28.5 GB | 29.8 GB | Won't fit | — | 32.9 |
| IQ1_M | 1.75 | 24.3 GB | 25.7 GB | Won't fit | — | 39.2 |