Sieve
Collection
Decision models that return calibrated probabilities for typed questions about text or JSON, without generating text. • 24 items • Updated • 10
Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
engine: string
engine_options: struct<model: string>
child 0, model: string
model_source: struct<model: string, source: string, base_model: string, temperature: double, serving: string, adap (... 61 chars omitted)
child 0, model: string
child 1, source: string
child 2, base_model: string
child 3, temperature: double
child 4, serving: string
child 5, adapter_model.safetensors_sha256: string
child 6, head.pt_sha256: string
torch: string
transformers: string
peft: string
gpu: string
loaded_seconds: double
frozen_corpus_sha256: string
rows_path: list<item: string>
child 0, item: string
runner_sha256: string
latency: string
edition: string
note: string
counts: struct<ok: int64>
child 0, ok: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6414 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6402 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: do
...
GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 3, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 4, iSarcasmEval-A-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 8, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'brier': Value('float64'), 'log_loss': Value('float64'), 'accuracy': V
...
lue('int64'), 'mean': Value('float64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
engine: string
engine_options: struct<model: string>
child 0, model: string
model_source: struct<model: string, source: string, base_model: string, temperature: double, serving: string, adap (... 61 chars omitted)
child 0, model: string
child 1, source: string
child 2, base_model: string
child 3, temperature: double
child 4, serving: string
child 5, adapter_model.safetensors_sha256: string
child 6, head.pt_sha256: string
torch: string
transformers: string
peft: string
gpu: string
loaded_seconds: double
frozen_corpus_sha256: string
rows_path: list<item: string>
child 0, item: string
runner_sha256: string
latency: string
edition: string
note: string
counts: struct<ok: int64>
child 0, ok: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6414 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6402 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: do
...
GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 3, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 4, iSarcasmEval-A-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 8, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'brier': Value('float64'), 'log_loss': Value('float64'), 'accuracy': V
...
lue('int64'), 'mean': Value('float64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Sieve-9B: Decision Index results
The complete run of Sieve-9B on the frozen suite of the Decision Index 0.2.1, made with the official kit
apolinario/decision-index at 87d4650 (Version 0.2.1); the kit's own scorer produced the scores.
| Leaderboard | Full score 41.73, public score 41.88 (edition 0.4.1, board data of 2026-10-26) |
| This run | Decision Index 0.2.1: 41.71 (raw index 56.47, breadth 40.06) |
| Requests | all 150,317 scoreable requests answered: 0 unsupported, 0 errors (150,759 rows with the 442 official exclusions) |
The Full score ranks the board: it comes from the maintainers' private tests, 60% tests of the same skills as the public benchmarks and 40% decision tasks from new domains.
| area | skill (×100) |
|---|---|
| Knowledge & Reasoning | 27.4 |
| Language Understanding | 46.1 |
| Retrieval & Classification | 46.1 |
| Tools & Automation | 61.3 |
| Arts & Human Taste | 22.8 |
Files (runs/Sieve-9B/)
| file | contents |
|---|---|
scores.json |
the kit's scores: "complete": true, edition 0.2.1, index, areas, per-benchmark values |
index.json, benchmark-summary.json |
per-benchmark raw, skill and coverage; native metrics and counts |
environment.json, status.json |
the runner's own environment and final status (complete, 150,759 rows, all ok) |
results.jsonl.gz |
one result per request: run id, status, the model's answers with a probability for every option, and timings |
results.jsonl.gz is in the runner's --compact form. The payload field (benchmark text) and the
raw_output field are left out, because several suite sources do not allow their text to be republished. Nothing else
differs. Scoring this file with python -m decision_index score --results runs/Sieve-9B/results.jsonl.gz reproduces
scores.json exactly, apart from its generation time. The run was recorded in a folder named sieve-9b and was
re-scored after the folder was renamed Sieve-9B, so scores.json names the run Sieve-9B. Every score is unchanged,
and results.jsonl.gz, environment.json and status.json are byte-identical to the runner's output.
How the run was made
- Suite. Rebuilt with
suite rebuildand staged withsuite import. The rows are byte-identical to the lab's (selected-rowsuncompressed sha256b2b56d6f…,added-rows7429f3c9…, exclusions331df32d…, subsets matched). HLE was fetched with an account that accepted its terms. Raw suite inputs are not redistributed. - Model.
sthanika-ai/Sieve-9B. That is a LoRA adapter plus a pointer head onQwen/Qwen3.5-9Batc202236, with the calibrated temperature stored inhead.pt. The weights this run loaded have adapter sha256f14e56eb…and head sha256c8a943e4…; both are recorded inenvironment.jsonand match the published files. - Engine.
sieve.decision_index_engine:SieveEngine, published at github.com/sthanika-ai/Sieve.- The run itself used an earlier copy of the same engine. The published code gives identical answers: 1,098 of 1,098 questions on 364 re-run requests from all 91 tracks, with the largest probability difference 0.
- The adapter is merged into the bf16 backbone, the pointer head runs in fp32, and the path is eager.
- A single-question request runs as one causal row: the state and the question together.
- A multi-question request prefills the state once, then answers the questions in batches that fit memory.
- Nothing is truncated, no option is dropped, and no prompt is changed per benchmark.
- Declared limits: a 65,536-token state, a 32,768-token question and up to 255 choice options. No request reached them.
- Hardware. One NVIDIA A100 80GB PCIe, shared with other users' jobs; single process, one request at a time. torch 2.8.0+cu128, transformers 5.17.0, peft 0.21.0, flash-linear-attention 0.5.2.
- Local latency. Median 69.3 ms per request, p95 692.8 ms, over 9 h 5 min of wall time on the shared GPU. This is not the maintainers' RTX PRO 6000 measurement.
- Downloads last month
- 199
