Jeeves – Reasoning improves Jev-like decision models

A Jev-style classifier that reasons when it is unsure, with a diffusion drafter, trained on public data.

Inspired by Kev.

• A 9B Jev-like model (Qwen3.5-9B, LoRA, pointer head) that reasons before it decides when it is unsure, with a block-4 diffusion drafter and the full training code and train/dev/test data.
• Reasoning only when unsure, it beats Kev-9B, Jev and JevK5 on test data it was never trained on (0.905 vs 0.822, 0.857 and 0.802) and on JevBench's public tiers (0.896 vs 0.866 for Jev and 0.861 for JevK5).
• Supports yes/no (noul), multiple-choice (choice), and rating (score) questions in the same request, through a Jev-compatible API.
• About 20 ms per request without thinking and a 1.2 s median with it on one H100 with --precision fp8. Can be sped up by truncating chain length.
• Runs on CUDA in bf16, or with FP8 linear layers (--precision fp8) on GPUs with compute capability 8.9 or higher. Inference also runs on Apple Silicon (MPS) in bf16 or FP8.

Jev-like models give calibrated decision probabilities, but at low accuracy. A lot of pipelines therefore rely on a reasoning model as a fallback. Jeeves trains a Jev-like Qwen3.5-9B (LoRA and a pointer head) to reason before it decides when it is unsure.

This results in better performance on out of domain tasks, and outperforms Jev in JevBench hard (public).

Jeeves v2 and v1 answer directly when their calibrated confidence is at least 0.9 and otherwise reason first (nothink_threshold 0.9, greedy, 2,560-token cap); the percentage is the share of questions that reasoned. JevK5 (v0.3) does not reason; we ran it ourselves on the same items. The Kev-9B and Jev columns are the numbers Kev publishes.

* No Kev-9B JevBench result is published. These are Kev-8B (Qwen3).

All JevBench numbers are on the public easy, standard and hard tiers (231 items). The sealed judge tier is not included, and the Jev and Kev numbers are restricted to the same public items.

Without thinking v2 scores 0.812 on our test split (2,962 items), against 0.851 reasoning when unsure. On held-out questions from the 15 public datasets new in v2 (maths, commonsense, tool choice and fact checking), it scores 0.853 and 0.900.

Requirements: Python 3.12 and a CUDA GPU. Inference also runs on an Apple Silicon Mac with 48 GB or more.

pip install -r requirements.txt

Download the released weights and serve them:

hf download PostHog/jeeves --local-dir jeeves-weights
python -m inference.serve --model jeeves-weights --drafter jeeves-weights/drafter_k4.safetensors --port 8009

On a Mac, the engine runs on MPS. The weights use 21 GB in bf16. The default caches (--max-rows 8 --max-len 8192) use another 28 GB, so a 48 GB Mac starts to swap. Use smaller caches:

python -m inference.serve --model jeeves-weights --drafter jeeves-weights/drafter_k4.safetensors --max-rows 4 --max-len 4096 --port 8009

On an M4 Pro, one question thinks at about 20 tokens per second, and a request without thinking takes 0.3 to 0.5 s.

--precision fp8 quantizes the linear layers as on CUDA and runs them with a Metal w8a16 kernel. Activations stay bf16 at every size, as on CUDA. The weights then use 11.5 GB, and one question thinks at about 38 tokens per second. The outputs change slightly; on dev questions, accuracy and NLL did not change measurably.

FP8 weights can be downloaded from PostHog/jeeves-fp8. They give the same outputs as --precision fp8 on the bf16 weights:

hf download PostHog/jeeves-fp8 --local-dir jeeves-fp8
python -m inference.serve --model jeeves-fp8 --drafter jeeves-fp8/drafter_k4.safetensors --precision fp8 --max-rows 4 --max-len 4096 --port 8009

python -m model.metal_test checks the Metal kernels against float64 and eager references. On CUDA, python -m model.triton_kernels_test checks the Triton rotary, delta-state advance, delta gates, conv step and cache writes bit for bit against the eager and fla paths, and the decode attention against float32. python -m inference.cuda_test checks the merged projections, the attention masks and the FP8 GEMM. python -m inference.engine_test checks how answer_batch packs requests into groups, on any device. python speed.py --model jeeves-weights --data data/dev.jsonl times a fixed set of dev requests; python -m prep.prep builds data/.

Or fuse your own trained checkpoint into a standalone model and serve it with a drafter:

python export.py runs/cispo/final --out runs/fused
python -m inference.serve --model runs/fused --drafter runs/drafter_k4/drafter.safetensors --port 8009

python export_fp8.py runs/fused --drafter runs/drafter_k4/drafter.safetensors --out runs/fused-fp8 writes the FP8 weights of an export and its drafters, in the format of PostHog/jeeves-fp8.

Then send a request in Jev's format:

curl -s localhost:8009/v1/systemone -H 'content-type: application/json' -d '{
  "state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
  "questions": {
    "department":  {"type": "choice", "instructions": "Which team should handle this?",
                    "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                 "shipping": "Delivery status, delays, lost packages",
                                 "billing": "Charges, invoices, payment problems"}},
    "escalate":    {"type": "noul", "instructions": "Does this need urgent human attention?"},
    "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                    "criteria": ["Calm", "Frustrated", "Very angry"]}
  },
  "options": {"max_think": 512}}'

Response on one H100 with --precision fp8, with the three questions thinking in parallel:

{
    "model": "jeeves-latest",
    "answers": {
        "department": {
            "type": "choice",
            "choice": "billing",
            "confidence": 0.19,
            "probabilities": { "returns": 0.4, "shipping": 0.14, "billing": 0.46 }
        },
        "escalate": { "type": "noul", "noul": 0.72 },
        "frustration": {
            "type": "score",
            "score": 1.5,
            "legend": { "0": "Calm", "1": "Frustrated", "2": "Very angry" },
            "probabilities": { "0": 0.04, "1": 0.43, "2": 0.54 },
            "confidence": 0.75
        }
    },
    "usage": { "input_tokens": 129, "output_tokens": 160, "reasoning_tokens": 1536 },
    "latency_ms": 8141.6
}

sdk/ is a drop-in replacement for Jev's Python SDK (typesafe-sdk):

pip install ./sdk

from jeeves_sdk import Choice, Noul, Score, TypeSafeClient

with TypeSafeClient() as client:
    result = client.system_one(
        state="I was charged twice. Please help.",
        questions={
            "billing": Noul(instructions="Is this about billing?"),
            "tone": Choice(instructions="What is the tone?", criteria={"calm": None, "angry": None}),
            "urgency": Score(instructions="How urgent is this?", criteria=["can wait", "this week", "today"]),
        },
        max_think=768,
        return_reasoning=True,
    )
    print(result.nouls["billing"].noul, result.choices["tone"].choice, result.scores["urgency"].score)
    print(result.reasoning["tone"].text)

The client connects to http://127.0.0.1:8009 by default (or JEEVES_BASE_URL), needs no API key, and waits up to 120s.

options is optional and ignored by Jev clients that don't send it. Server-wide defaults are set with the matching serve flags.

On 325 dev questions, on one H100 with --precision fp8:

v2 was trained on chains of at most 1,024 tokens, so lower caps cut most of its chains. These use v1's drafter.

Questions, states and answers are loaded into the Qwen chat template like

<state> …state…
<q> instructions <opt> option 1 </opt> <opt> option 2 </opt> …
<think>

The model then rolls out its reasoning chain, and after the </think> token we append

</think>

<q> instructions <opt> option 1 </opt> <opt> option 2 </opt> …
<decide>

A pointer head scores each option with a scaled dot product between a query projection of the hidden state at <decide> and a key projection of the hidden state at that option's </opt>, where

<state>, <q>, <opt>, </opt>, <decide> = "<|fim_prefix|>", "<|fim_middle|>", "<|box_start|>", "<|box_end|>", "<|fim_suffix|>"

These are rare, largely unused tokens in the Qwen tokenizer. Ablations found that using plain text like "State" in the prompt instead worsened performance. Likewise, not repeating the questions after the reasoning block also decreases performance. The final probabilities are a softmax over the option scores, divided by a temperature fitted on the dev set. Answers after reasoning get their own temperature.

1. SFT (1 epoch, 16,867 steps on 4 GPUs). LoRA r=16 on all projections of Qwen3.5-9B plus the pointer head, trained on 453,531 questions from about 175 public sources: public datasets, the public OpenJev and Kev decision sets, third-party public decision datasets labelled by teacher models, prompt-injection sets and v1's data. About 15% of examples carry a reasoning chain sampled from earlier rounds of the model, kept only if it closes within 1,024 tokens, does not loop and reaches the right answer.
2. Calibration. Two temperatures fitted on dev and stored with the checkpoint: 1.52 for direct answers and 5.42 for answers after reasoning. Every training chain reaches the right answer, so answers after reasoning come out overconfident and need the larger temperature.

v1 was trained with SFT followed by CISPO. CISPO did not improve v2.

A diffusion view of the frozen model (drafter/), inspired by Orthrus.

Unlike Orthrus, which supports attention-only models, it supports Qwen3.5's Gated DeltaNet layers by letting mask tokens cross-attend to those layers' post-convolution keys and values.

Block 4 is the default because it stays cheap when several questions are batched.

You can build the datasets locally using the prep scripts. This downloads the public datasets at pinned revisions and writes train/dev/rl in the order listed in prep/manifests/v2/, checking their sha256:

python -m prep.prep

Each public dataset stays under its own license.

On 8 GPUs, with the data in data/, bash run.sh runs the whole v1 pipeline:

torchrun --nproc_per_node 8 train.py sft --run-dir runs/sft
torchrun --nproc_per_node 8 train.py cispo --run-dir runs/cispo --init runs/sft/final
torchrun --nproc_per_node 8 test.py runs/cispo/final
torchrun --nproc_per_node 8 jevbench.py runs/cispo/final
python export.py runs/cispo/final --out runs/fused
torchrun --nproc_per_node 8 -m drafter.gen --model runs/fused
torchrun --nproc_per_node 8 train.py drafter --model runs/fused --block 4 --run-dir runs/drafter_k4

• Knowledge questions trail Jev (MMLU 0.845 vs 0.900, MMLU-Pro 0.770 vs 0.840).
• Thinking is slow at the tail: 8 s at p90 with full chains on one H100. Use max_think and nothink_threshold when latency matters.
• The Kev and Jev comparisons outside JevBench use different items from the same sources.
• No language consistency reward was included so thinking chains are not well interpretable.
• v2 trails v1 on JevBench when reasoning only when unsure (0.896 vs 0.913): it is overconfident without reasoning on some long-policy and multi-hop items, so it skips reasoning on them. With reasoning on every item the two are level (0.922 vs 0.926).
• The reasoning chains used for SFT are not published yet, so prep.prep rebuilds the training records but not the chains.

If you use Jeeves, its training recipe or its drafter, please cite:

@software{waltz2026jeeves,
  author = {Waltz, Nicholas P.},
  title  = {Jeeves: Reasoning Improves Jev-like Decisions},
  year   = {2026},
  url    = {https://github.com/PostHog/jeeves},
  note   = {Qwen3.5-9B decision model trained with SFT and CISPO, with a block-4 diffusion drafter}
}

• Jev's Architecture Unmasked, the Jev design that Kev and Jeeves follow.
• Qwen Team. Qwen3.5-9B, the base model.
• Yang, Kautz, Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.
• Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
• MiniMax. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. 2025. Introduces CISPO.
• Guo, Pleiss, Sun, Weinberger. On Calibration of Modern Neural Networks. ICML 2017. Temperature scaling.
• Orthrus, arXiv 2605.12825. The diffusion drafter ours adapts to Gated DeltaNet.
• Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023.