Gemma 4 31B-it QAT assistant (drafter), exact Q4_0 GGUF (unofficial)

This is an unofficial build, made and published independently of Google. It is a llama.cpp GGUF of Google's speculative-decoding drafter for Gemma 4 31B-it QAT, google/gemma-4-31B-it-qat-q4_0-unquantized-assistant, with every weight matrix stored at 4 bits on the grid quantization-aware training produced. It is 279,956,288 B, against 514,706,240 B for the Q8_0 file llama.cpp's converter writes. Google's model card is kept unchanged in ORIGINAL_README.md.

Use it with a Gemma 4 31B-it QAT target, for example google/gemma-4-31B-it-qat-q4_0-gguf. Google's card says the drafter must match the target's QAT precision.

The 31B target is 17.7 GB. On an 8 GB card, 20 of its 60 layers fit beside this drafter and the rest run on the CPU, which is the setting measured here:

llama-server -m gemma-4-31B_q4_0-it.gguf -ngl 20 -fa 1 -c 8192 -t 8 \
  -md gemma-4-31B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 1

Measured on llama.cpp build 7182ef71e only.

Per prompt at 1 drafted token: facts 1.71x, code 1.70x, explanation 1.58x, chat 1.55x, story 1.51x, summary 1.47x. The drafter adds 362 MiB of VRAM (nvidia-smi: 7,020 MiB without, 7,382 with).

With most of the target on the CPU, use 1 drafted token. Checking 2–4 drafted tokens at a time costs the CPU-side target more than checking 1, so the extra accepted tokens do not pay for themselves here. With the whole target on a GPU the best length is likely different; that was not measured.

Measured on an RTX 3050 OEM (8 GB) in an i7-5820K desktop (6 cores, quad-channel DDR4, 46 GiB), llama.cpp 7182ef71e, target google/gemma-4-31B-it-qat-q4_0-gguf with -ngl 20 -t 8, flash attention, f16 KV cache, -c 8192: six prompts (code, chat, story, explanation, facts, summary), 128 tokens, greedy, one run per prompt (the target generates at 2.5 tok/s), geometric mean over prompts. No drafter ran first and last, at 2.5 tok/s both times.

Greedy text changes. With the drafter, greedy replies differ from the target's greedy replies without a drafter in 4 of 6 at 1 drafted token. The target still checks every drafted token; the differences likely come from rounding in the batched check (inferred, not measured). Answer quality with this drafter has not been measured on the 31B; for the 12B's drafter, built the same way, GSM8K and BFCL scored level with and without it.

The Q8_0 conversion was not speed-tested against this file on the 31B. For the 12B drafter this format was 1.06x the Q8_0 one at the same acceptance, and for the 26B A4B drafter 1.03x.

What is in it

• Built from google/gemma-4-31B-it-qat-q4_0-unquantized-assistant at revision 96d4c8c: 4 transformer layers of width 1024 (3 sliding-window, 1 full attention), with queries only; keys and values come from the target's cache. Two projections join it to the target's 5376-wide hidden state. The output head is the tied 262,144-token embedding.
• Every weight matrix as Q4_0 (the embedding, 4 x attention q/o, 4 x MLP gate/up/down, both projections), each block's trained step recovered: largest magnitude divided by 8, 7, … 1, whichever puts all 32 values on an integer level, refined by least squares and stored as the fp16 block scale. No block was off the grid. 96.6–97.1% of values per tensor reproduce the bf16 source exactly once rounded back to bf16.
• Metadata (hyperparameters, tokenizer, chat template) is llama.cpp's converter output, general.file_type set to Q4_0. Norms and scales stay F32.
• Text only. The drafter has no vision or audio parts.

Files

evidence/ holds the build script (gguf_exact.py), its per-tensor log and report (build.log, build-report.json), the speed run's script and per-prompt table (spec_ab.py, speed-summary.txt) and the converter log. speed-summary.txt labels its cells "median of 3"; this run had one repeat per cell.

License

Apache 2.0, as Google's source. Gemma 4 terms: https://ai.google.dev/gemma/docs/gemma_4_license