Qwen-AgentWorld-35B-A3B, readout fine-tuned (LoRA, + coherence)

A System One decision model: it reads a state, answers typed questions (choice, score, noul) and returns calibrated probability distributions your code can branch on; it never writes text. This repo is a rank-16 LoRA (8,355,840 parameters) on Qwen/Qwen-AgentWorld-35B-A3B (35B parameters, about 3B active per token), trained on its own decision readout with a coherence penalty, the same method as the other jevify-*-readout-coh models.

At a glance (jev-bench, 22,773 test records) — accuracy 0.770 · ECE 0.055 · held-out 0.828 · TVD to human labels 0.305 · sure loss 0.030
same order, the untuned base model (Tier 0): 0.709 / 0.077 / 0.793 / 0.408 / 0.138
same order, Jev 1.13.0: 0.733 / 0.113 / 0.835 / 0.432 / 0.081
same order, Perplexity's pplx-decider-v1-27b (hosted API, as served): 0.748 / 0.080 / 0.840 / 0.378 / 0.219
on ten public benchmarks it was not trained on (macro accuracy, 4,850 questions): this model 83.2 · untuned base 80.0 · Perplexity 86.1 · Jev 87.3. See below.

Each panel has its own axis; the strip under each panel gives this model's difference to each competitor. One training run (one seed); Perplexity is the hosted API, as served.

jev-bench · leaderboard · Perplexity comparison · code

One run, one seed. The numbers below come from a single training run; the largest gain over the untuned model (+6.1 points of accuracy) is far outside seed noise on the other backbones we trained (±0.4 to ±0.8 points), but nothing here is a seed average.

Use it: serve it with vLLM

Verified path: vLLM 0.30 with the adapter loaded as a LoRA on the base weights (no merged copy is needed). The adapter is saved twice: lora/ has the tensor names Hugging Face transformers produced (model.layers.N...); lora_vllm/ has the same tensors renamed to model.language_model.layers.N..., which vLLM requires. vLLM does not complain about the first form: it loads it ("Loaded new LoRA adapter") and silently ignores every weight, so the served model is the untuned base. Always compare an adapter-served answer with a base-served one after loading.

vllm serve Qwen/Qwen-AgentWorld-35B-A3B --served-model-name agentworld-base --tensor-parallel-size 2 --dtype bfloat16 \
  --max-model-len 8192 --max-logprobs 260 --enable-prefix-caching \
  --enable-lora --max-lora-rank 16 --max-loras 1 --lora-modules agentworld-jevify=./lora_vllm \
  --limit-mm-per-prompt '{"image":0,"video":0}' --gpu-memory-utilization 0.93 --max-num-seqs 32 \
  --disable-custom-all-reduce          # two PCIe GPUs without peer-to-peer; VLLM_USE_FLASHINFER_SAMPLER=0 on a box without a CUDA toolkit

Those flags are what two 46 GB L40S needed; the memory-share and sequence limits leave room for LoRA's profiling pass. Not tested: loading the adapter with transformers + peft directly (everything here was served through vLLM).

The readout is the Jevify Tier 0 readout over the OpenAI-compatible /v1/completions endpoint: the question, the allowed answers and an Answer: cue are rendered into the model's chat template with thinking off; one forward pass; the log-probabilities of the allowed answer tokens at the cue are read (max_tokens=1, logprobs=260, allowed_token_ids, return_tokens_as_token_ids=true) and the recipe is applied. serve/client.py does all of it (pip install the engine for the prompt renderer):

from client import decide
decide({"text": "The battery lasted two days."}, {"type": "noul", "instructions": "Is the review positive?"})

The recipe (jevify_config.json; fitted on jev-bench validation, nothing from the test split): choice probabilities averaged over two option orders, a temperature per question type (1.085 choice, 1.175 score, 1.146 yes/no) and a yes/no bias of +0.3 on the log-odds. For comparison the untuned base model needed 1.54 / 3.64 / 2.07 and a bias of −1.0: RL-post-trained models read out over-confidently, and training the readout removes most of that.

Three traps we hit while serving (each produced plausible-looking output):

1. The adapter is silently ignored if its tensors are named for the wrong wrapper (above).
2. vLLM 0.30's --logprobs-mode processed_logprobs returned a one-hot here (one answer at 0, the rest at −9999); the default raw mode is used instead. With many options a few allowed answers can fall outside the raw top-260 (11 of 151 in one CLINC150 example) and the client gives them a floor just below the lowest value returned (they are below a log-probability of about −13).
3. Option identifiers must be single tokens. With more than 26 options, two-letter codes that split into two tokens collide with other candidates; the client uses the same single-token rule as the engine.

Results

Every number is on the jev-bench test splits (22,773 records), scored with the repo's own code, macro-averaged over the 22 tasks. "Held-out" = the six tasks that no fine-tuning here trained on (clinc150, arc_challenge, yelp5, measuring_hate_speech, fever_evidence, strategyqa_grounded).

The sure loss is 0.052 on choice families, 0.009 on yes/no families and 0.023 on rating families, below Jev's 0.069 / 0.032 / 0.157 on all three. results/ holds the full per-task metrics, the recipe, the training record and the coherence analysis; results/test_metrics.json has every task.

What fine-tuning did, task by task (accuracy, untuned → this model): HelpSteer2 verbosity 27.7 → 66.0, measuring_hate_speech 48.5 → 69.5, go_emotions 27.8 → 49.5, civil_comments 78.3 → 92.2, strategyqa_closed 65.5 → 79.5, stsb 48.4 → 56.0. It cost accuracy on two rating scales: yelp5 62.3 → 52.2 (held out) and sst5 56.7 → 52.2, and ChaosNLI 67.1 → 65.6: the same effect as in the other models (training on other rating scales moves the level boundaries of an unseen one; a per-level bias fitted on 16 labelled reviews restores most of it for Qwen3.5-9B).

Latency (vLLM, one request at a time, two L40S, prefix caching off, measured on the untuned weights; a LoRA adds no layers): 50 ms at 129 input tokens, 62 ms at 312, 95 ms at 912, 129 ms at 1,659, 388 ms at 7,001. A request has a fixed cost of about 44 ms plus about 46 µs per input token. Thirty-two simultaneous short requests complete in 0.6 s (19 ms per question). Prefix caching helps only when a shared prefix exceeds one 1,056-token block (this hybrid model's block size).

Serving check. Through the vLLM server above, client.py reproduced the training run's own test predictions: on 48 choice questions from five tasks it gave the same answer as the offline fine-tuned model on 47 (and it differs from the base-served answer on 4, so the adapter is active); on 40 yes/no questions the probability of yes differed by 0.006 on average (largest 0.051); on 40 rating questions the expected level differed by 0.09 levels on average (largest 0.29). The offline predictions come from the merged weights in bf16, the served ones from an unmerged LoRA, so small differences are expected.

External public benchmarks (not trained on)

jev-bench is the benchmark this model was fine-tuned for, so it flatters the model. To see where it stands elsewhere, the same questions were put to four systems on ten public benchmark families that were not in the fine-tuning data. Perplexity's own panel items are not published, so we built the ten families from their public sources, uniformly, with a fixed seed and up to 500 items each (4,850 questions); every system answered exactly the same questions. Nothing was fitted on these items: this model uses its jev-bench recipe unchanged.

Accuracy (%). Jev and Perplexity are the hosted APIs as served; "untuned" is the base model with its own fitted recipe.

Intervals are paired-bootstrap 95% (items resampled within benchmark, 2,000 resamples); one benchmark of 500 items is good to about ±4 points. "n.s." = the interval includes zero. Macro ECE: Jev 0.043, Perplexity 0.060, this model 0.056, untuned 0.059.

What it shows:

1. On independent benchmarks this model is behind both systems: 4.1 points behind Jev and 2.8 behind Perplexity (macro). Its lead on jev-bench is an in-distribution result, and should be read that way.
2. The fine-tuning transfers. On benchmarks it never trained on it is +3.2 points over the untuned base, significantly better on five (Financial PhraseBank +11.2, BBH +5.2, RAGTruth +5.2, JudgeBench +5.1, TabFact +4.6) and not significantly worse on any.
3. The gap is reasoning. It is concentrated on BBH (−21.8 against Jev), WinoGrande (−11.4), TabFact (−8.2) and TruthfulQA (−8.0): items a model normally works out by writing a chain of thought, which a one-pass readout of a ~3B-active model does not do, and which training the readout does not add.
4. Where it is ahead or level: Financial PhraseBank (ahead of both), ContractNLI (ahead of Jev), RAGTruth, JudgeBench and Belebele (level with Jev).
5. Calibration is not a lead here (macro ECE 0.056 against Jev's 0.043); on jev-bench it was (0.055 against 0.113).
6. The model knows when it is unsure, which makes a hand-off useful: its confidence ranks its own right answers above its wrong ones with AUROC 0.825, and its most confident 50% of answers are 96.3% correct. Handing the 20% least-confident questions to Perplexity lifts the macro from 83.2% to 85.9% (to Jev: 86.5%); sending everything to Perplexity gives 86.1%.

Caveats: these are our constructions, not the items of Perplexity's panel, so the numbers compare the systems with each other, not with the numbers on Perplexity's card. Circa is the 5-way goldstandard2 labelling; Financial PhraseBank is the 100%-annotator-agreement version (easier than 50%); long inputs were not truncated for any system (the vLLM server ran with a 16,384-token context). None of the ten was used in fine-tuning; whether the base model saw their public texts in pretraining is unknown. Rebuild, rerun and every system's per-question predictions are in eval/public_panel/.

How it was trained

Readout fine-tuning with a coherence penalty (FINDINGS 18): the loss is a proper scoring rule on the restricted answer distribution at the cue (log score for choice and yes/no, ranked probability score for ratings) plus β = 1 times the squared distance of each training question's family of related questions to its nearest coherent answer set (the sure loss). LoRA rank 16 on the shared-expert projections of all 40 layers and q/k/v/o of the 10 full-attention layers (160 modules; the routed experts and the Gated DeltaNet projections are not adapted), learning rate 3e-5, two epochs, the epoch with the lower validation loss kept (epoch 1 of 2: validation loss 0.466 against 0.473), 5,885 training questions from the 16 non-held-out jev-bench tasks (up to 400 per task), seed 0, bf16; 4.4 hours on three L40S.

Limits

One seed. On independent public benchmarks it is behind Jev and Perplexity, most of all on reasoning-heavy ones (BBH, WinoGrande, TabFact). Text only (the checkpoint has no vision-tower weights, so image input is disabled when serving). The coherence penalty covers the relations the family generator builds (a choice and its options, a yes/no and its negation, a rating and its thresholds), not every logical relation. Do not use it for decisions where an error is costly without checking its probabilities on your own data first.

Files