Hugging Face's logo

Datasets:
sthanika-ai
/
Sieve-0.8B-decision-index-results

Tags:
decision-index
evaluation
decision-model
License:
Dataset card Files
xet
 Community
Dataset Viewer
Duplicate
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code:   StreamingRowsError
Exception:    CastError
Message:      Couldn't cast
engine: string
engine_options: struct<sieve_dir: string, mem_fraction: string, max_batch_tokens: int64>
  child 0, sieve_dir: string
  child 1, mem_fraction: string
  child 2, max_batch_tokens: int64
model_source: struct<kind: string, path: string, device: string, dtype: string, base_model: string, base_revision: (... 227 chars omitted)
  child 0, kind: string
  child 1, path: string
  child 2, device: string
  child 3, dtype: string
  child 4, base_model: string
  child 5, base_revision: string
  child 6, list_options: bool
  child 7, built_in_temperature: double
  child 8, context_limit_tokens: int64
  child 9, full_weights.safetensors_sha256: string
  child 10, head.safetensors_sha256: string
  child 11, sieve_engine.py_sha256: string
  child 12, modeling_sieve.py_sha256: string
note_on_package: string
torch: string
transformers: string
flash-linear-attention: string
causal-conv1d: string
suite: struct<edition: string, selected_rows_sha256: string, added_rows_sha256: string, gsm8k_rows_sha256:  (... 48 chars omitted)
  child 0, edition: string
  child 1, selected_rows_sha256: string
  child 2, added_rows_sha256: string
  child 3, gsm8k_rows_sha256: string
  child 4, exclusions_sha256: string
  child 5, note: string
execution: struct<part_1: struct<rows: string, kit: string, hardware: string, mode: string, started_utc: timest (... 202 chars omitted)
  child 0, part_1: struct<rows: string, kit: string, hardware: string, mode: string, started_utc: timestamp[s], finishe (... 2
...
equests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
              child 0, metric: string
              child 1, score: double
              child 2, scored_requests: int64
          child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
              child 0, metric: string
              child 1, score: double
              child 2, scored_requests: int64
counts: struct<ok: int64, unsupported: int64>
  child 0, ok: int64
  child 1, unsupported: int64
edition: string
note: string
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
  child 0, median: double
  child 1, p95: double
  child 2, mean: double
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
                  raise CastError(
                  ...<3 lines>...
                  )
              datasets.table.CastError: Couldn't cast
              engine: string
              engine_options: struct<sieve_dir: string, mem_fraction: string, max_batch_tokens: int64>
                child 0, sieve_dir: string
                child 1, mem_fraction: string
                child 2, max_batch_tokens: int64
              model_source: struct<kind: string, path: string, device: string, dtype: string, base_model: string, base_revision: (... 227 chars omitted)
                child 0, kind: string
                child 1, path: string
                child 2, device: string
                child 3, dtype: string
                child 4, base_model: string
                child 5, base_revision: string
                child 6, list_options: bool
                child 7, built_in_temperature: double
                child 8, context_limit_tokens: int64
                child 9, full_weights.safetensors_sha256: string
                child 10, head.safetensors_sha256: string
                child 11, sieve_engine.py_sha256: string
                child 12, modeling_sieve.py_sha256: string
              note_on_package: string
              torch: string
              transformers: string
              flash-linear-attention: string
              causal-conv1d: string
              suite: struct<edition: string, selected_rows_sha256: string, added_rows_sha256: string, gsm8k_rows_sha256:  (... 48 chars omitted)
                child 0, edition: string
                child 1, selected_rows_sha256: string
                child 2, added_rows_sha256: string
                child 3, gsm8k_rows_sha256: string
                child 4, exclusions_sha256: string
                child 5, note: string
              execution: struct<part_1: struct<rows: string, kit: string, hardware: string, mode: string, started_utc: timest (... 202 chars omitted)
                child 0, part_1: struct<rows: string, kit: string, hardware: string, mode: string, started_utc: timestamp[s], finishe (... 2
              ...
              equests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: double
                            child 2, scored_requests: int64
                        child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: double
                            child 2, scored_requests: int64
              counts: struct<ok: int64, unsupported: int64>
                child 0, ok: int64
                child 1, unsupported: int64
              edition: string
              note: string
              successful_request_latency_ms: struct<median: double, p95: double, mean: double>
                child 0, median: double
                child 1, p95: double
                child 2, mean: double
              to
              {'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
              ...
              Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
              because column names don't match

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

Sieve-0.8B: Decision Index results

The complete run of Sieve-0.8B on the public suite of the Decision Index 0.3, made with the official kit apolinario/decision-index at 87d4650 (0.2.1 requests) and 62d2f51 (rebuilt GSM8K requests); the kit's own scorer produced the scores.

Leaderboard On the Decision Index leaderboard, edition 0.4.1 (board data of 2026-10-26): Full score 20.82
This run Decision Index 0.3: public index 32.26 (raw 49.40, breadth 28.90)
Requests 140,178: 140,176 answered, 2 unsupported (options with identical text), 0 errors

Leaderboard score and this run

The two numbers on this card measure different things:

  • Full score 20.82 is the leaderboard's ranking score. It comes from the maintainers' private tests: 60% tests of the same skills as the public benchmarks and 40% decision tasks from new domains. Only the maintainers run the private parts, so this card contains no per-request data for them.
  • Public index 32.26 is our own run of the public 0.3 suite with the official kit. It is the public part alone, which is why it is higher than the Full score. The files in this dataset are this run.

The board's public suite (edition 0.4.1) has 110,201 requests over 37 benchmarks, while this run uses the 0.3 suite (140,178 scoreable requests over 43 benchmarks, 37 of them in the index), so the request counts differ.

area skill (×100)
Knowledge & Reasoning 14.3
Language Understanding 26.6
Retrieval & Classification 54.8
Tools & Automation 51.2
Arts & Human Taste 13.5

Files (runs/Sieve-0.8B/)

file contents
scores.json the kit's scores: "complete": true, edition 0.3, index, areas, per-benchmark values
index.json, benchmark-summary.json per-benchmark raw, skill and coverage; native metrics and counts
environment.json, status.json engine, options, package hashes, versions, suite hashes, and how the run was executed
results.jsonl.gz one result per request: run id, status, the model's answers with a probability for every option, timings

results.jsonl.gz is in the runner's --compact form: payload (benchmark text) and raw_output are left out, because several suite sources do not allow their text to be republished. Every row keeps its payload_sha256. Scoring this file with python -m decision_index score --edition 0.3 --results runs/Sieve-0.8B/results.jsonl.gz reproduces scores.json.

How the run was made

  • Suite. Rebuilt with suite rebuild; the 0.2.1 rows, the rebuilt GSM8K rows and the exclusions match the kit's pinned hashes, and suite verify --edition 0.3 passes. Raw suite inputs are not redistributed.
  • Model. sthanika-ai/Sieve-0.8B, full fine-tuned blocks 1-16 and a scalar head on Qwen/Qwen3.5-0.8B-Base at dc7cdfe2, bf16, built-in temperature 1.1208, context limit 32,768 tokens. Engine sieve_engine:SieveEngine, then shipped in the model repo and now sieve.fork.engine.ForkEngine in github.com/sthanika-ai/Sieve (run it with --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-0.8B), options max_batch_tokens=49152 and a per-process mem_fraction cap (0.14 / 0.12 / 0.25). Nothing is truncated, no option is dropped, no prompt is changed per benchmark.
  • Execution, in two parts with the same engine, package, runner and pinned versions:
    1. The 0.2.1 suite on 2× A100 80GB PCIe (shared machine): 24 round-robin shards of exactly the rows pipeline runs, 8 processes (5 on GPU 0, 3 on GPU 1), merged by run id with no duplicates and no retries.
    2. The 2,638 rebuilt GSM8K requests of 0.3 on the same machine, one process, appended to that file exactly as resuming it under 0.3 would.
  • Latency (single process, one request at a time, bf16, A100 80GB, 400 uniform suite requests): median 69 ms, mean 92 ms, 80th percentile 91 ms. The latency_ms in scores.json was recorded in the shared multi-process run. The leaderboard measures latency separately, on one NVIDIA RTX PRO 6000.
Downloads last month
157
Total file size:
44.7 MB

Collection including sthanika-ai/Sieve-0.8B-decision-index-results