Gemma · Gemma 3 · 12.2B parameters

Gemma 3 12B VRAM requirements

Gemma 3 12B has 48 layers and uses grouped-query attention (8 KV heads). At Q4_K_M the weights come to 7.0 GB, and the best quantisation that fits a 24 GB card is INT8 / W8A8.

Model

Hardware

Quantisation

Context 8K

KV cache precision

Runtime

Batch

GPUs

Monitor plugged into this GPU

Runs comfortably

8.6 GB of 21.8 GB · 39%

022 GB

Weights 7.0 GB

KV cache 0.8 GB

Runtime overhead 0.8 GB

Gemma 3 12B at Q4_K_M leaves 13.2 GB spare on a RTX 4090. There is room to raise the context length or move up a quantisation level.

Generation78.5tok/s

Prompt processing2840tok/s

Max context128Ktokens

KV per 1K tokens0GB

Every quantisation of Gemma 3 12B on a RTX 4090

Highlighted row is the highest quality that still fits at 8K context.

Quantisation	bpw	Weights	Total	Verdict	Max ctx	tok/s
FP16 / BF16	16.00	22.7 GB	24.4 GB	Won't fit	—	9.09
INT8 / W8A8	8.50	11.8 GB	13.5 GB	Runs comfortably	128K	48.0
Q8_0 (GGUF)	8.50	11.8 GB	13.5 GB	Runs comfortably	128K	48.0
FP8 (E4M3)	8.00	11.4 GB	13.0 GB	Runs comfortably	128K	50.0
Q6_K	6.56	9.3 GB	10.9 GB	Runs comfortably	128K	60.1
Q5_K_M	5.67	8.1 GB	9.7 GB	Runs comfortably	128K	68.8
Q5_K_S	5.52	7.8 GB	9.5 GB	Runs comfortably	128K	70.5
AWQ 4-bit	4.25	7.4 GB	9.0 GB	Runs comfortably	128K	74.2
GPTQ 4-bit	4.25	7.4 GB	9.0 GB	Runs comfortably	128K	74.2
MXFP4	4.25	7.4 GB	9.0 GB	Runs comfortably	128K	74.2
Q4_K_M	4.85	7.0 GB	8.6 GB	Runs comfortably	128K	78.5
Q4_K_S	4.58	6.6 GB	8.2 GB	Runs comfortably	128K	82.3
Q4_0	4.55	6.6 GB	8.2 GB	Runs comfortably	128K	82.7
IQ4_XS	4.25	6.2 GB	7.8 GB	Runs comfortably	128K	87.4
Q3_K_M	3.91	5.7 GB	7.4 GB	Runs comfortably	128K	93.3
IQ3_M	3.70	5.5 GB	7.1 GB	Runs comfortably	128K	97.5
IQ3_XXS	3.06	4.6 GB	6.3 GB	Runs comfortably	128K	113
Q2_K	2.63	4.1 GB	5.7 GB	Runs comfortably	128K	126
IQ2_XXS	2.06	3.3 GB	5.0 GB	Runs comfortably	128K	149
IQ1_M	1.75	2.9 GB	4.6 GB	Runs comfortably	128K	165

Gemma 3 12B on each GPU

Q4_K_M weights at 8K context, single card, monitor attached.

GPU	VRAM	GB/s	Verdict	Max ctx	tok/s
H100 SXM 80GB	80	3350	Runs comfortably	128K	286
A100 80GB	80	2039	Runs comfortably	128K	162
RTX 5090	32	1792	Runs comfortably	128K	151
RTX 5080	16	960	Runs comfortably	98K	82.3
RTX 4090	24	1008	Runs comfortably	128K	78.5
RTX 5070 Ti	16	896	Runs comfortably	98K	76.9
RTX 3090	24	936	Runs comfortably	128K	76.1
Radeon RX 7900 XTX	24	960	Runs comfortably	128K	71.1
L40S	48	864	Runs comfortably	128K	67.5
RTX A6000	48	768	Runs comfortably	128K	62.7
RTX 3080 10GB	10	760	Fits, but tight	8K	62.1
Mac Studio M3 Ultra 256GB	256	819	Runs comfortably	128K	61.8
RTX 5070	12	672	Runs comfortably	38K	58.0
RTX 4080 Super	16	736	Runs comfortably	98K	57.7
RTX 4070 Ti Super	16	672	Runs comfortably	98K	52.7
Mac Studio M4 Max 128GB	128	546	Runs comfortably	128K	46.1
RTX 4070 Super	12	504	Runs comfortably	38K	39.7
RTX 4070	12	504	Runs comfortably	38K	39.7
RTX 5060 Ti 16GB	16	448	Runs comfortably	98K	38.9
Arc B580	12	456	Runs comfortably	38K	30.5
RTX 3060 12GB	12	360	Runs comfortably	38K	29.7
NVIDIA DGX Spark (GB10)	128	273	Runs comfortably	128K	23.8
Mac Mini M4 Pro 48GB	48	273	Runs comfortably	128K	23.2
RTX 4060 Ti 16GB	16	288	Runs comfortably	98K	22.8
Ryzen AI Max+ 395 128GB	128	256	Runs comfortably	128K	17.8

Architecture

Parameters	12.2B
Layers	48
Hidden size	3840
Attention heads / KV heads	16 / 8
Head dimension	256
Vocabulary	262,144
Trained context	128K
Sliding window	1024 (every 6th layer is global)
KV cache per 1K tokens	0 GB
Hugging Face	google/gemma-3-12b-it

The Gemma family

Google's open models. Two things dominate the memory: a 262k vocabulary — on Gemma 3 1B the embedding table is about a third of the file — and sliding-window attention, where only every sixth layer of Gemma 3 sees the full context, and every second layer on Gemma 2. The cache grows far more slowly than the context length suggests.

huggingface.co/google · ai.google.dev/gemma · all 6 Gemma models

Gemma 3 1B
1B · 26 layers Gemma 3 4B
4.3B · 34 layers Gemma 3 27B
27.4B · 62 layers Gemma 2 9B
9.2B · 42 layers Gemma 2 27B
27.2B · 46 layers

Direct answers

Gemma 3 12B on RTX 5090
See the verdict Gemma 3 12B on RTX 4090
See the verdict Gemma 3 12B on RTX 3090
See the verdict Gemma 3 12B on RTX 5080
See the verdict Gemma 3 12B on RTX 5070 Ti
See the verdict Gemma 3 12B on RTX 5070
See the verdict Gemma 3 12B on RTX 4070 Ti Super
See the verdict Gemma 3 12B on RTX 4070 Super
See the verdict Gemma 3 12B on RTX 4070
See the verdict