Gemma 4 E2B-it QAT assistant (drafter), exact Q4_0 GGUF (unofficial)

This is an unofficial build, made and published independently of Google. It is a llama.cpp GGUF of Google's speculative-decoding drafter for Gemma 4 E2B-it QAT, google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant, with every weight matrix stored at 4 bits on the grid quantization-aware training produced. It is 59,236,000 B, against 97,836,704 B for the Q8_0 file llama.cpp's converter writes, and follows Google's weights more closely. Google's model card is kept unchanged in ORIGINAL_README.md.

Use it with a Gemma 4 E2B-it QAT target, for example xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf or google/gemma-4-E2B-it-qat-q4_0-gguf. Google's card says the drafter must match the target's QAT precision.

llama-server -m <E2B QAT Q4_0 target>.gguf -ngl 99 -fa 1 \
  -md gemma-4-E2B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2

Needs a llama.cpp with Gemma 4 MTP support (#23398, June 2026). Measured on builds 77c4540ce (RTX 3050) and f95b0d9 (GTX 1650 Ti).

Speed is level with the Q8_0 file and acceptance is the same. The gain is size, memory and fidelity. Google's bf16 weights, served unchanged, accept the same share of tokens (0.475) and run slower (133.1 tok/s, 1.23x), so drafter precision does not limit acceptance. Without a drafter the target runs at 108.2 tok/s.

Measured on an RTX 3050 OEM (8 GB), llama.cpp 77c4540ce, target xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf, flash attention, f16 KV cache: six prompts (code, chat, story, explanation, facts, summary), 256 tokens, greedy, median of three runs, geometric mean over prompts; 2 drafted tokens per step. Every drafter precision produced the same greedy replies.

On a GTX 1650 Ti (4 GB laptop GPU)

E2B target xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf, llama.cpp f95b0d9, same prompts and method, 1 drafted token per step (best on this card):

Without a drafter the target runs at 78.9 tok/s in 1,485 MiB. With this drafter at 1 / 2 / 3 drafted tokens: 1.18x / 1.11x / 1.02x (RTX 3050: 1.40x at 2 tokens). Acceptance is about the same on both cards, so the smaller gain likely comes from each drafted step costing a larger share of a target step on this card (inferred, not measured). Use --spec-draft-n-max 1 on a card of this class.

On a GTX 1650 Ti, llama.cpp 77c4540ce

Same card on the RTX 3050's build, same prompts and method. Without a drafter the target runs at 76.5 tok/s; with this drafter at 1 drafted token, 90.3 tok/s (1.18x), 0.602 accepted.

1 drafted token is best on this build; use --spec-draft-n-max 1 here. With a long text in the context, this drafter at 1 token gives 1.27x / 1.18x / 1.03x / 0.96x / 0.89x at 0 / 8k / 32k / 64k / 120k tokens ahead (-c 131072); on the RTX 3050 it kept paying to 120k.

Answer quality, 8 requests at once (-c 32768 --parallel 8), greedy:

Every difference has a 95% bootstrap range that includes zero. With 8 requests at once the drafter slows the work on this card (on the RTX 3050 it sped the same work up 1.5-1.7x); it pays for one request at a time.

Answer quality with the drafter serving

The target model checks every drafted token, so the drafter changes speed and leaves the target's choices to the target, apart from rounding in the batched check. E2B target, llama.cpp 77c4540ce, 8 requests at once, greedy, 2 drafted tokens:

Paired on the same problems, every drafted arm's difference from no drafter has a 95% bootstrap range that includes zero.

What is in it

• Built from google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant at revision ebc7e1a: 4 transformer layers of width 256 (3 sliding-window, 1 full attention), with queries only; keys and values come from the target's cache. Two projections join it to the target's 1536-wide hidden state. The output head is the tied 262,144-token embedding.
• Every weight matrix as Q4_0 (the embedding, 4 x attention q/o, 4 x MLP gate/up/down, both projections), each block's trained step recovered: largest magnitude divided by 8, 7, … 1, whichever puts all 32 values on an integer level, refined by least squares and stored as the fp16 block scale. No block was off the grid. 96.5–97.1% of values per tensor reproduce the bf16 source exactly once rounded back to bf16.
• Metadata (hyperparameters, tokenizer, chat template) is llama.cpp's converter output, general.file_type set to Q4_0. Norms and scales stay F32.
• Left out by llama.cpp's converter: the checkpoint's masked_embedding centroids and token ordering, a clustered shortcut for the output head. llama.cpp scores the full vocabulary instead.
• Text only. The drafter has no vision or audio parts.

Files

evidence/ holds the build script (gguf_exact.py), its per-tensor report, the RTX 3050 speed run's per-prompt table (speed-summary.txt) and the GTX 1650 Ti tables (speed-summary-1650ti-precision.txt, speed-summary-1650ti-draft-length.txt, and on 77c4540ce speed-summary-1650ti-77c4540ce.txt, longctx-summary-1650ti-77c4540ce.txt).

License

Apache 2.0, as Google's source. Gemma 4 terms: https://ai.google.dev/gemma/docs/gemma_4_license