Sieve
Collection
Decision models that return calibrated probabilities for typed questions about text or JSON, without generating text. • 24 items • Updated • 10
Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
edition: string
panel_id: string
index: double
raw_index: double
scores: struct<balanced_skill: double, balanced_raw: double, breadth_skill: double>
child 0, balanced_skill: double
child 1, balanced_raw: double
child 2, breadth_skill: double
areas: list<item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, (... 32 chars omitted)
child 0, item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, benchmarks (... 20 chars omitted)
child 0, id: string
child 1, label: string
child 2, raw: double
child 3, skill: double
child 4, coverage: double
child 5, n: int64
child 6, benchmarks: list<item: int64>
child 0, item: int64
benchmarks: struct<1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_inde (... 4909 chars omitted)
child 0, 1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 6, tracks: list<item: null>
child 0, item: null
child 1, 2: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, covera
...
le
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 39, 24: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 40, 26: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 41, 27: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 42, 34: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
coverage: double
note: string
engine: string
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
counts: struct<ok: int64>
child 0, ok: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float64')}, 'positive_micro_f1': Valu
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
edition: string
panel_id: string
index: double
raw_index: double
scores: struct<balanced_skill: double, balanced_raw: double, breadth_skill: double>
child 0, balanced_skill: double
child 1, balanced_raw: double
child 2, breadth_skill: double
areas: list<item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, (... 32 chars omitted)
child 0, item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, benchmarks (... 20 chars omitted)
child 0, id: string
child 1, label: string
child 2, raw: double
child 3, skill: double
child 4, coverage: double
child 5, n: int64
child 6, benchmarks: list<item: int64>
child 0, item: int64
benchmarks: struct<1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_inde (... 4909 chars omitted)
child 0, 1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 6, tracks: list<item: null>
child 0, item: null
child 1, 2: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, covera
...
le
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 39, 24: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 40, 26: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 41, 27: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 42, 34: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
coverage: double
note: string
engine: string
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
counts: struct<ok: int64>
child 0, ok: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float64')}, 'positive_micro_f1': Valu
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Sieve-27B: Decision Index results
The complete run of Sieve-27B on the public suite of the Decision Index 0.3, made with the official kit
apolinario/decision-index at 62d2f51 (0.3); the kit's own scorer produced the scores.
| Leaderboard | Full score 60.17, public score 60.04 (edition 0.3.2, board data of 2026-10-12); no longer listed since edition 0.4 |
| This run | Decision Index 0.3: public index 60.04 (raw index 69.32, breadth 58.67) |
| Requests | all 140,178 scoreable requests answered: 0 unsupported, 0 errors |
In edition 0.3.2 the Full score counted the public benchmarks 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.
| area | skill (×100) |
|---|---|
| Knowledge & Reasoning | 43.5 |
| Language Understanding | 65.7 |
| Retrieval & Classification | 67.8 |
| Tools & Automation | 77.5 |
| Arts & Human Taste | 40.8 |
Files (runs/Sieve-27B/)
| file | contents |
|---|---|
scores.json |
the kit's scores: "complete": true, edition 0.3, index, areas, per-benchmark values |
index.json, benchmark-summary.json |
per-benchmark raw, skill and coverage; native metrics and counts |
shard-0/, shard-1/ |
each runner's environment and final status (70,310 + 70,310 requests, all ok) |
results.jsonl.gz |
one result per request: run id, status, the model's answers with a probability for every option, and timings |
RUN_NOTES.md |
how the run was made, step by step, with checksums |
results.jsonl.gz is in the runner's --compact form. The payload field (benchmark text) and the raw_output field
are left out, because several suite sources do not allow their text to be republished.
Scoring this file with
python -m decision_index score --edition 0.3 --results runs/Sieve-27B/results.jsonl.gz reproduces scores.json
exactly, apart from its generation time.
How the run was made
- One complete run of the 0.3 suite, in two shards, one per A100 80GB PCIe (2026-10-08), merged into one file.
Details and checksums are in
RUN_NOTES.md. - Suite. Rebuilt with
suite rebuildand staged withsuite import, with every hash verified. Raw suite inputs are not redistributed. - Model.
sthanika-ai/Sieve-27B, a LoRA adapter plus a pointer head onQwen/Qwen3.8-27Bat1d4bf0f2, with the calibrated temperature (1.050) stored inhead.pt. The run loaded adapter sha25662cc35bb…and head sha2560c8551e5…; both are recorded in everyenvironment.jsonand match the published files. - Engine.
sieve.decision_index_engine:SieveEnginefrom github.com/sthanika-ai/Sieve, with--option model=<local copy of Sieve-27B>and--compact.- The adapter is merged into the bf16 backbone, the pointer head runs in fp32, and the path is eager.
- Nothing is truncated, no option is dropped, and no prompt is changed per benchmark.
- Declared limits: a 65,536-token state, a 32,768-token question and up to 255 choice options. No request reached them. The model was trained on states and questions of up to 16,384 tokens each; 241 requests had a question longer than that (at most 29,944 tokens, from ToolRet and BRIGHT) and were answered in full. No state was longer than 16,384 tokens.
- Software. Torch 2.8.0+cu128, transformers 5.17.0, peft 0.21.0, flash-linear-attention 0.5.2, triton 3.4.0 and causal-conv1d 1.7.0.
- Local latency (not the maintainers' measurement). One request at a time, eager, over the 140,620 requests of the run. The median is 151 ms, the mean 416 ms, the 80th percentile 425 ms and the 95th percentile 966 ms.
- Downloads last month
- 229
