Sieve
Collection
Decision models that return calibrated probabilities for typed questions about text or JSON, without generating text. • 24 items • Updated • 10
Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
edition: string
panel_id: string
index: double
raw_index: double
scores: struct<balanced_skill: double, balanced_raw: double, breadth_skill: double>
child 0, balanced_skill: double
child 1, balanced_raw: double
child 2, breadth_skill: double
areas: list<item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, (... 32 chars omitted)
child 0, item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, benchmarks (... 20 chars omitted)
child 0, id: string
child 1, label: string
child 2, raw: double
child 3, skill: double
child 4, coverage: double
child 5, n: int64
child 6, benchmarks: list<item: int64>
child 0, item: int64
benchmarks: struct<1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_inde (... 4909 chars omitted)
child 0, 1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 6, tracks: list<item: null>
child 0, item: null
child 1, 2: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, covera
...
le
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 39, 24: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 40, 26: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 41, 27: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 42, 34: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
coverage: double
note: string
counts: struct<ok: int64>
child 0, ok: int64
engine: string
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float64')}, 'positive_micro_f1': Valu
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 149, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 129, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 544, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 3052, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2559, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2584, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 558, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 440, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 361, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2381, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2309, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
edition: string
panel_id: string
index: double
raw_index: double
scores: struct<balanced_skill: double, balanced_raw: double, breadth_skill: double>
child 0, balanced_skill: double
child 1, balanced_raw: double
child 2, breadth_skill: double
areas: list<item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, (... 32 chars omitted)
child 0, item: struct<id: string, label: string, raw: double, skill: double, coverage: double, n: int64, benchmarks (... 20 chars omitted)
child 0, id: string
child 1, label: string
child 2, raw: double
child 3, skill: double
child 4, coverage: double
child 5, n: int64
child 6, benchmarks: list<item: int64>
child 0, item: int64
benchmarks: struct<1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_inde (... 4909 chars omitted)
child 0, 1: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 6, tracks: list<item: null>
child 0, item: null
child 1, 2: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool, t (... 24 chars omitted)
child 0, raw: double
child 1, skill: double
child 2, covera
...
le
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 39, 24: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 40, 26: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 41, 27: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
child 42, 34: struct<raw: double, skill: double, coverage: double, random: double, rule: string, in_index: bool>
child 0, raw: double
child 1, skill: double
child 2, coverage: double
child 3, random: double
child 4, rule: string
child 5, in_index: bool
coverage: double
note: string
counts: struct<ok: int64>
child 0, ok: int64
engine: string
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float64')}, 'positive_micro_f1': Valu
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Sieve-31B-v3: Decision Index results
The complete run of Sieve-31B-v3 (a full fine-tune of Gemma-4-31B-it with bidirectional attention in its 10 global layers, read through its own LM head at the option labels) on the public suite of the Decision Index 0.3, made with the official kit
apolinario/decision-index at f5a6e4f (0.3); the kit's own scorer produced the scores.
| Leaderboard | ranked #1 (tied): Full score 71.10, public score 71.68 (edition 0.4.1, board data of 2026-10-28) |
| This run | Decision Index 0.3: public index 71.68 (raw index 78.39, breadth 71.18) |
| Requests | all 140,178 scoreable requests answered: 0 unsupported, 0 errors, nothing truncated |
| Latency (1× RTX PRO 6000, one engine per GPU) | median 219 ms, mean 913 ms, 80th percentile 657 ms |
The board's Full score, which ranks the models, comes from the maintainers' private tests: private tests of the same skills (60%) and private tasks from new domains (40%). The public index above is our own run of the public suite.
| area | skill (×100) |
|---|---|
| Knowledge & Reasoning | 61.2 |
| Language Understanding | 71.6 |
| Retrieval & Classification | 76.1 |
| Tools & Automation | 86.9 |
| Arts & Human Taste | 62.3 |
Files (runs/Sieve-31B-v3/)
| file | contents |
|---|---|
scores.json |
the kit's scores: "complete": true, edition 0.3, index, areas, per-benchmark values |
index.json, benchmark-summary.json |
per-benchmark raw, skill and coverage; native metrics and counts |
shard-0/ … shard-7/ |
the runner's environment and final status for each eighth of the run (17,577 or 17,578 requests each, all ok) |
results.jsonl.gz |
one result per request: run id, status, the model's answers with a probability for every option, and timings |
RUN_NOTES.md |
how the run was made, step by step, with checksums |
results.jsonl.gz is in the runner's --compact form. The payload field (benchmark text) and the raw_output field
are left out, because several suite sources do not allow their text to be republished.
Scoring this file with
python -m decision_index score --edition 0.3 --engine sieve.decision_index_engine:SieveEngine --results runs/Sieve-31B-v3/results.jsonl.gz
reproduces scores.json, index.json and benchmark-summary.json exactly, apart from their generation time.
How the run was made
- One complete 0.3 run (2026-10-27), on eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. The 140,620 rows
of the 0.3 suite were split into eight shards of 17,577 or 17,578 rows, balanced by the suite's proxy token counts.
One runner ran on each GPU, and the eight
results.jsonlfiles were concatenated. The file has 140,620 rows; the 442 official exclusions leave 140,178 scored. - Suite.
suite-0.3from the kit (rows sha256b2b56d6f…, added rows7429f3c9…, rebuilt GSM8K75933489…, all inscores.json). Raw suite inputs are not redistributed. - Model.
sthanika-ai/Sieve-31B-v3, a full fine-tune ofgoogle/gemma-4-31B-itat842da379(the linear projections of its 60 decoder layers), stored as a full bf16 model plushead.pt. The released weights are a weighted average of the base model (0.3) and the fine-tuned model (0.7).head.ptholds the readout ("letters"), the attention rule, the serving policy and the calibration temperatures: a global temperature (1.27) and one per question category, 22 in all. The run loadedhead.ptsha25675b2073d…, recorded in everyenvironment.json. Before the run, the weights,head.pt, configuration and tokenizer files on the GPU host were checked against their SHA-256; they are the files published in the model repository (listed in itsprovenance.json). - Engine.
sieve.decision_index_engine:SieveEnginefrom github.com/sthanika-ai/Sieve (0.4.0, which adds the letter-readout enginesieve.letters), with--option model=<local copy of Sieve-31B-v3>,--rows <shard>and--compact.- The answer of each question is the model's own LM-head logits of the option labels at the answer slot, softmaxed at the question category's temperature.
- In the 10 global layers, the shared part of a request (system turn, the list of questions, the state) attends only to itself and each question attends to that part and to itself; the 50 sliding-window layers are causal. In a request with several questions and a shared part of 2,048 tokens or more, the shared part is computed once and its keys and values are reused by every question, which gives the same result as computing it with each question.
- Every question whose options may be reordered is read twice, with its options as given and rotated by half; the two log-probability vectors are averaged.
- bf16 weights, eager PyTorch. Nothing is truncated, no option is dropped, and no prompt is changed per benchmark. Declared limits: no token limit and up to 255 choice options; every request fit.
- Software. Python 3.12, torch 2.8.0+cu128, transformers 5.17.0, huggingface_hub 1.32.0, safetensors 0.8.0 and accelerate 1.15.0.
- Latency (local, not the maintainers' measurement). In-process wall time of one request, CUDA-synchronized, eager, over all 140,620 requests, with one engine per RTX PRO 6000: the median is 219 ms, the mean 913 ms, the 80th percentile 657 ms and the 95th percentile 1,507 ms.
- Downloads last month
- 54
