Gemma 4 E4B-it QAT, exact Q4_0 GGUF, text only (unofficial)

This is an unofficial build, made and published independently of Google. It is a llama.cpp GGUF of Gemma 4 E4B-it in which every 4-bit weight is the value Google's quantization-aware training (QAT) produced. Every weight matrix is stored at 4 bits, including both embedding tables. It is 4.22 GB, against 5.15 GB for Google's gemma-4-E4B-it-qat-q4_0-gguf, and follows the unquantized model much more closely. Google's model card is kept unchanged in ORIGINAL_README.md.

The E2B counterpart, built by the same script, is xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf.

Measured with llama.cpp build f95b0d9 (318) on an i7-10750H laptop with a GTX 1650 Ti Max-Q (4 GB, no tensor cores). Divergence and perplexity: llama-perplexity on wikitext-2 test, 16 chunks of 512 tokens, on the CPU, against a bf16 GGUF of the same QAT weights with identical metadata. Speed: llama-bench, 5 repetitions, four passes alternating which file runs first, each pass starting at or below 50 °C; the table gives the mean of the four passes, and the generation ratio was 1.058–1.063 in every pass. Logs are in evidence/.

It fits a 4 GB card. With -ngl 99 -c 8192 the whole process holds 2848 MiB, because per_layer_token_embd (1.59 GB, 38% of the file) is read a few rows per token from the memory-mapped file and never goes to VRAM. The KV cache at 8192 context is 168 MiB.

What is in it

• Language model only. Vision and audio live in Google's separate gemma-4-E4B-it-mmproj.gguf, which is not included here.
• Metadata copied byte for byte from Google's GGUF at revision 4b4a2c1: tokenizer, chat template, hyperparameters and tensor order.
• Weights rebuilt from google/gemma-4-E4B-it-qat-q4_0-unquantized at revision 476025a: all 342 layer matrices (attention, MLP, per-layer gate and projection), both embedding tables (token_embd, per_layer_token_embd) and per_layer_model_proj, as Q4_0.
• Copied unchanged: the norms and scale vectors (F32, 2.2 MB in all).

Why it is smaller and closer

QAT trained these weights on a 4-bit grid: in every block of 32 values, each value is a block scale times an integer from -8 to 7. That holds for the two embedding tables and per_layer_model_proj as well as the layers. The embedding tables make up 3.49 billion of the model's values, and Google's GGUF stores them as Q6_K. Stored as Q4_0 they keep the trained values and take less space, which is where most of the 0.94 GB goes.

For the layer matrices, Google's GGUF uses llama.cpp's standard Q4_0 step, the block's largest magnitude divided by 8. QAT often puts a block's largest value on a lower level, and in those blocks the standard step moves values off the trained grid. This build recovers each block's trained step instead (largest magnitude divided by 8, 7, … 1, whichever puts every value on an integer level, then refined by least squares) and stores it as the fp16 block scale.

Every one of the rebuilt blocks landed on a 4-bit grid, and 96.81% of rebuilt values are bit-identical to the source after bf16 rounding. The rest differ by the fp16 rounding of the scale. Per-tensor counts are in evidence/build_report.json.

The generation speed-up is about half the E2B build's (1.11–1.12x on the same card). That fits where the bytes are: token_embd doubles as the output projection and is read in full for every token, and it is a smaller share of what E4B reads per token than of what E2B does. Memory traffic was not measured directly.

Running it

llama.cpp (tested on CUDA; divergence measured on CPU):

llama-server -m gemma-4-E4B-it-q4_0-exact.gguf -ngl 99 -fa 1 -c 8192 --jinja   # GPU
llama-server -m gemma-4-E4B-it-q4_0-exact.gguf -t 6 -c 8192 --jinja            # CPU

Do not pass --no-mmap: llama.cpp reads per_layer_token_embd lazily from the map, and without it the file would not fit a 4 GB card.

llama-server turns thinking on by default, so a reply lands in reasoning_content and a short max_tokens can leave content empty. To get direct answers, start the server with --reasoning off, or send "chat_template_kwargs": {"enable_thinking": false} with a request.

Rebuilding it

hf download google/gemma-4-E4B-it-qat-q4_0-unquantized --revision 476025a --local-dir src
hf download google/gemma-4-E4B-it-qat-q4_0-gguf gemma-4-E4B_q4_0-it.gguf --revision 4b4a2c1 --local-dir gguf
python3 gguf_exact.py src gguf/gemma-4-E4B_q4_0-it.gguf gemma-4-E4B-it-q4_0-exact.gguf

Needs only numpy, and takes about 7.5 minutes and 6.2 GiB of RAM on a laptop. The output is deterministic: SHA-256 5462cc102b7d22c4748f9b8e82091f2ae064e10a45525151af6f9a7d0da7caf9. --bf16 writes the reference instead: the same file with the source bf16 weights. The script stops if any block fails to land on a 4-bit grid. It is the E2B build's script, unchanged apart from its docstring.

Limitations

• Text only.
• Tested with llama.cpp on one laptop: divergence on its CPU (AVX2), speed and five greedy chat prompts on its GPU (GTX 1650 Ti, CUDA). Other backends and runtimes were not tested, and divergence was not measured on CUDA.
• Built and checked on one machine only; the E2B build was reproduced on two.
• The divergence figures cover 8,192 tokens of English Wikipedia text.
• Unofficial. Report problems here, not to Google.

License and attribution

Gemma 4 is released by Google DeepMind under the Apache 2.0 license. This repository redistributes Google's weights in a changed storage format, under the same license. The weights, training, tokenizer, chat template and the original model card are Google DeepMind's; this build and card are not affiliated with or endorsed by Google.