Cohere on NVIDIA Ada

Can I run Command R 35B on an RTX 4090?

Not at Q4_K_M — it needs 31.0 GB against 21.8 GB available. You would need 2 of these cards.

Won't fit

31.0 GB of 21.8 GB · 142%
032 GB
Weights 20.1 GB
KV cache 10.0 GB
Runtime overhead 0.9 GB
Over the limit 9.2 GB

Short by 9.2 GB. You can run it with 21 of 40 layers on the RTX 4090 and the rest in system RAM, at roughly 3.00 tok/s — usable for batch work, painful for chat. A smaller quantisation or a shorter context is usually the better trade.

Generation3.00tok/s
Prompt processing991tok/s
Max context640tokens
KV per 1K tokens0GB

Every quantisation of Command R 35B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

QuantisationbpwWeights TotalVerdictMax ctxtok/s
FP16 / BF16 16.00 65.2 GB 76.0 GB Won't fit — 0.63
INT8 / W8A8 8.50 33.7 GB 44.6 GB Won't fit — 1.38
Q8_0 (GGUF) 8.50 33.7 GB 44.6 GB Won't fit — 1.38
FP8 (E4M3) 8.00 32.6 GB 43.5 GB Won't fit — 1.46
Q6_K 6.56 26.7 GB 37.6 GB Won't fit — 1.93
Q5_K_M 5.67 23.1 GB 34.0 GB Won't fit — 2.35
AWQ 4-bit 4.25 23.0 GB 33.9 GB Won't fit — 2.36
GPTQ 4-bit 4.25 23.0 GB 33.9 GB Won't fit — 2.36
MXFP4 4.25 23.0 GB 33.9 GB Won't fit — 2.36
Q5_K_S 5.52 22.5 GB 33.4 GB Won't fit — 2.51
Q4_K_M 4.85 20.1 GB 31.0 GB Won't fit 640 3.00
Q4_K_S 4.58 19.1 GB 30.0 GB Won't fit 1K 3.27
Q4_0 4.55 19.0 GB 29.9 GB Won't fit 2K 3.29
IQ4_XS 4.25 17.9 GB 28.8 GB Won't fit 2K 3.81
Q3_K_M 3.91 16.7 GB 27.6 GB Won't fit 3K 4.50
IQ3_M 3.70 15.9 GB 26.8 GB Won't fit 4K 4.96
IQ3_XXS 3.06 13.7 GB 24.5 GB Won't fit 6K 7.47
Q2_K 2.63 12.1 GB 23.0 GB Won't fit 7K 12.4
IQ2_XXS 2.06 10.1 GB 21.0 GB Fits, but tight 9K 39.7
IQ1_M 1.75 9.0 GB 19.8 GB Fits, but tight 10K 42.9

Also worth checking

Command R 35B on every GPU
Full hardware tableEverything that runs on a RTX 4090
Full model tableWhy the KV cache is the part that bites
Guide