Sieve
Collection
Decision models that return calibrated probabilities for typed questions about text or JSON, without generating text. • 24 items • Updated • 10
Dataset Viewer
The dataset viewer is not available because its heuristics could not detect any supported data files. You can try uploading some data files, or configuring the data files location manually.
Sieve-31B: Decision Index results
The complete run of Sieve-31B on the public suite of the Decision Index 0.3, made with the official kit
apolinario/decision-index at 62d2f51 (0.3); the kit's own scorer produced the scores.
| Leaderboard | Full score 64.25 on the Decision Index leaderboard, edition 0.4.1 (board data of 2026-10-26); no longer listed since 2026-10-28 |
| This run | Decision Index 0.3: public index 62.52 (raw 71.08, breadth 61.55) |
| Requests | 140,178 of 140,178 scored requests ok: 0 unsupported, 0 errors, nothing truncated |
Leaderboard score and this run
The two numbers on this card measure different things:
- Full score 64.25 was the leaderboard's ranking score. It comes from the maintainers' private tests: 60% tests of the same skills as the public benchmarks and 40% decision tasks from new domains. Only the maintainers run the private parts, so this card contains no per-request data for them.
- Public index 62.52 is our own run of the public 0.3 suite with the official kit. It is the public part alone. The files in this dataset are this run.
The board's public suite (edition 0.4.1) has 110,201 requests over 37 benchmarks, while this run uses the 0.3 suite (140,178 scoreable requests over 43 benchmarks, 37 of them in the index), so the request counts differ.
| area | skill (×100) |
|---|---|
| Knowledge & Reasoning | 49.9 |
| Language Understanding | 65.6 |
| Retrieval & Classification | 69.5 |
| Tools & Automation | 78.7 |
| Arts & Human Taste | 43.5 |
Files (runs/Sieve-31B/)
| file | contents |
|---|---|
scores.json |
the kit's scores: "complete": true, edition 0.3, index, areas, per-benchmark values |
index.json, benchmark-summary.json |
per-benchmark raw, skill and coverage; native metrics and counts |
results.jsonl.gz |
one result per request: run id, status, the model's answers with a probability for every option |
RUN_NOTES.md |
how the run was made, the published engine's parity with it, latency and checksums |
parity/ |
the RTX PRO 6000 parity and latency run: environment, status and comparison report |
code/ |
the batched evaluator, its engine classes, the temperature re-read and the scoring script, as used for this run |
The results contain only request ids, answers and probabilities: no suite payloads and no raw model output.
How the run was made
- Run. On 2026-10-10 on 2× NVIDIA A100 80GB PCIe, with a batched evaluator that uses the engine's prompts, readout
and refusal rules and packs many question rows into one forward pass. Probabilities were stored at T = 1 and
re-read at T = 8.5495 before scoring. Details are in
runs/Sieve-31B/RUN_NOTES.md. - Engine.
sieve.decision_index_engine:SieveEnginefrom github.com/sthanika-ai/Sieve with--option model=sthanika-ai/Sieve-31Bruns the model one request at a time; its parity with this run is inRUN_NOTES.md. - Latency (published engine, 1× RTX PRO 6000 Blackwell Server Edition, single process, eager, 400 random public rows): median 163 ms, mean 644 ms, 80th percentile 675 ms. The leaderboard measures latency separately, also on one NVIDIA RTX PRO 6000, and lists a median of 253 ms.
- Downloads last month
- 261
