Sieve
Collection
Decision models that return calibrated probabilities for typed questions about text or JSON, without generating text. • 24 items • Updated • 10
Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
edition: string
panel_id: string
index: double
raw_index: double
scores: struct<balanced_skill: double, balanced_raw: double, breadth_skill: double>
child 0, balanced_skill: double
child 1, balanced_raw: double
child 2, breadth_skill: double
areas: list<item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, (... 32 chars omitted)
child 0, item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, benchmarks (... 20 chars omitted)
child 0, id: string
child 1, label: string
child 2, raw: double
child 3, skill: double
child 4, coverage: double
child 5, n: int64
child 6, benchmarks: list<item: int64>
child 0, item: int64
benchmarks: struct<1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_inde (... 4909 chars omitted)
child 0, 1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 6, tracks: list<item: null>
child 0, item: null
child 1, 2: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, covera
...
le
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 39, 24: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 40, 26: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 41, 27: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 42, 34: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
coverage: double
note: string
counts: struct<ok: int64>
child 0, ok: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
engine: string
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float64')}, 'positive_micro_f1': Valu
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
edition: string
panel_id: string
index: double
raw_index: double
scores: struct<balanced_skill: double, balanced_raw: double, breadth_skill: double>
child 0, balanced_skill: double
child 1, balanced_raw: double
child 2, breadth_skill: double
areas: list<item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, (... 32 chars omitted)
child 0, item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, benchmarks (... 20 chars omitted)
child 0, id: string
child 1, label: string
child 2, raw: double
child 3, skill: double
child 4, coverage: double
child 5, n: int64
child 6, benchmarks: list<item: int64>
child 0, item: int64
benchmarks: struct<1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_inde (... 4909 chars omitted)
child 0, 1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 6, tracks: list<item: null>
child 0, item: null
child 1, 2: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, covera
...
le
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 39, 24: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 40, 26: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 41, 27: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 42, 34: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
coverage: double
note: string
counts: struct<ok: int64>
child 0, ok: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
engine: string
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float64')}, 'positive_micro_f1': Valu
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Sieve-9B-v2: Decision Index results
The complete run of Sieve-9B-v2 on the public suite of the Decision Index 0.3, made with the official kit
apolinario/decision-index at 62d2f51 (0.3); the kit's own scorer produced the scores.
| Leaderboard | Full score 47.00, public score 53.35 (edition 0.4.1, board data of 2026-10-26) |
| This run | Decision Index 0.3: public index 53.35 (raw index 64.39, breadth 51.95) |
| Requests | all 140,178 scoreable requests answered: 0 unsupported, 0 errors |
The Full score ranks the board: it comes from the maintainers' private tests, 60% tests of the same skills as the public benchmarks and 40% decision tasks from new domains.
| area | skill (×100) |
|---|---|
| Knowledge & Reasoning | 38.3 |
| Language Understanding | 55.9 |
| Retrieval & Classification | 64.2 |
| Tools & Automation | 69.8 |
| Arts & Human Taste | 34.0 |
Files (runs/Sieve-9B-Plus/)
The run was made when the model was named Sieve-9B-Plus, so the run folder and the files in it use that name.
| file | contents |
|---|---|
scores.json |
the kit's scores: "complete": true, edition 0.3, index, areas, per-benchmark values |
index.json, benchmark-summary.json |
per-benchmark raw, skill and coverage; native metrics and counts |
shard-0/ … shard-3/ |
the runner's environment and final status for each quarter of the run (35,155 requests each, all ok) |
results.jsonl.gz |
one result per request: run id, status, the model's answers with a probability for every option, and timings |
RUN_NOTES.md |
how the run was made, step by step, with checksums |
results.jsonl.gz is in the runner's --compact form. The payload field (benchmark text) and the raw_output field
are left out, because several suite sources do not allow their text to be republished.
Scoring this file with
python -m decision_index score --edition 0.3 --engine sieve.decision_index_engine:SieveEngine --results runs/Sieve-9B-Plus/results.jsonl.gz
reproduces scores.json, index.json and benchmark-summary.json exactly, apart from their generation time.
How the run was made
- One complete 0.3 run (2026-10-08), on two NVIDIA H100 80GB GPUs. The 140,620 rows of the 0.3 suite were split
into four shards of 35,155 rows, balanced by the suite's proxy token counts. Two runners shared each GPU, and the
four
results.jsonlfiles were concatenated. The file has 140,620 rows; the 442 official exclusions leave 140,178 scored. - Suite.
suite-0.3, staged with the kit and every hash verified (rows sha256b2b56d6f…, added rows7429f3c9…, rebuilt GSM8K75933489…, all inscores.json). Raw suite inputs are not redistributed. - Model.
sthanika-ai/Sieve-9B-v2, a LoRA adapter plus a pointer head onQwen/Qwen3.5-9Batc2022362. Itshead.ptstores a global temperature (1.109) and one temperature per question category, 23 in all. The run loaded adapter sha2564bb50d78…and head sha256c29fc83f…; both are recorded in everyenvironment.jsonand match the published files. - Engine.
sieve.decision_index_engine:SieveEnginefrom github.com/sthanika-ai/Sieve (0.2.0, which applies the per-category temperatures), with--option model=<local copy of Sieve-9B-Plus>,--rows <shard>and--compact.- The adapter is merged into the bf16 backbone, the pointer head runs in fp32, and the path is eager.
- Nothing is truncated, no option is dropped, and no prompt is changed per benchmark.
- Declared limits: a 65,536-token state, a 32,768-token question and up to 255 choice options. No request reached them. The model was trained on states and questions of up to 16,384 tokens each; 241 requests had a question longer than that (at most 29,944 tokens, from ToolRet and BRIGHT) and were answered in full. No state was longer than 16,384 tokens.
- Software. Torch 2.8.0+cu128, transformers 5.17.0, peft 0.21.0, flash-linear-attention 0.5.2, triton 3.7.1 and causal-conv1d 1.7.0.
- Local latency (not the maintainers' measurement). Measured on one H100 80GB GPU, single process, eager and in-process, over a random 3,000-request sample of the public suite. The median is 43 ms, the mean 138 ms, the 80th percentile 58 ms and the 95th percentile 144 ms. The answers on that sample matched this run exactly.
- Downloads last month
- 255
