Sieve-27B

Sieve-27B is a decision model from the Sieve family. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a LoRA adapter (rank 32) and a pointer head on Qwen/Qwen3.8-27B (revision 1d4bf0f2). The code is at github.com/sthanika-ai/Sieve.

Summary

How it works

• Backbone. Only the text part of Qwen3.8-27B is used; the model cannot produce text.
• Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the probabilities.
• Isolation. Each question runs as its own row from the state's cache, so questions never see each other.

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused kernel our runs used

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-27B")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

• Loading. Load it with sieve; the adapter alone is not a text-generation model. The backbone is downloaded at its pinned revision on first use.
• GPU memory. About 55 GB in bf16 (52,683 MiB in use after loading), so one 80 GB GPU is enough.
• Token limits. States and questions up to 16,384 tokens each, as in training. sieve 0.2.1 or later uses these limits by default; earlier versions stop at 8,192 tokens. Longer requests are refused, never truncated. The Decision Index engine declares larger limits (a 65,536-token state and a 32,768-token question). In the Decision Index run below, 241 requests had a question longer than 16,384 tokens (at most 29,944, from ToolRet and BRIGHT) and were answered in full; no state was longer than 16,384.
• Server. sieve-serve --model sthanika-ai/Sieve-27B --graphs.
• Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-27B --out runs/Sieve-27B.
• causal-conv1d. Without it, transformers falls back to a slower PyTorch convolution, whose probabilities can differ slightly. Install it to reproduce our numbers.
• Temperature. head.pt stores T = 1.050, fitted by NLL on the calibration split (43,309 questions). T never changes an answer, and temperature=1.0 gives raw probabilities.

Results

Decision Index

On the Decision Index leaderboard (edition 0.3.2, board data of 2026-10-12), Sieve-27B had a Full score of 60.17 and a public score of 60.04; it is no longer listed since edition 0.4. Our own run of the public 0.3 suite, with the official kit at 62d2f51 (0.3) and its scorer, scores 60.04 (raw index 69.32, breadth 58.67).

• Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
• How the run was made. One complete run of the 0.3 suite on two A100 80GB PCIe GPUs, in two shards.
• Results. sthanika-ai/Sieve-27B-decision-index-results.
• Full score. In edition 0.3.2 the Full score counted the public benchmarks 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

Calibration

The calibration split chose the checkpoint and fitted the temperature.

Latency

• Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of the 0.3 run (A100 80GB PCIe): median 151 ms, mean 416 ms, 80th percentile 425 ms, 95th percentile 966 ms. This is not the maintainers' RTX PRO 6000 measurement.
• Cached state. Server-reported p50 on one A100 80GB PCIe (bf16, merged, CUDA graphs), for one choice question on a 1,433-token state:

• By option count. For one choice question on a 41-token state, which is below the server's 64-token caching threshold, so every request ran the state again. Server p50 in ms, CUDA graphs / eager:

Training

• Recipe.
  • LoRA r=32, α=64, dropout 0.05 on the attention, MLP and linear-attention projections (12 modules), with the pointer head trained from scratch.
  • AdamW, lr 5e-5 for the adapter and the head, OneCycle with 10% warm-up, 1 epoch, 32 records per step, bf16.
  • States and questions up to 16,384 tokens each.
  • 4,813 steps, about 27 hours on two A100 80GB PCIe GPUs.
  • Permuted choice options and "none of the above" augmentation, with a cross-entropy loss.
• Selection. The checkpoint with the highest accuracy on a fixed 2,001-question calibration sample, with NLL breaking ties. This rule was fixed before training, and it chose step 4,500. The benchmarks were never used to choose.
• Data. 153,988 training records, with 30,776 for calibration and 123,177 for test (50/10/40).
  • Sources. Public datasets and generated decision records.
  • Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Files

License

Apache-2.0, see LICENSE.