Hugging Face's logo

Datasets:
sthanika-ai
/
Sieve-2B-v2-decision-index-results

Tags:
decision-index
evaluation
decision-model
License:
Dataset card Files
xet
 Community
Dataset Viewer
Duplicate
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code:   StreamingRowsError
Exception:    CastError
Message:      Couldn't cast
engine: string
engine_options: struct<sieve_dir: string, mem_fraction: double, max_batch_tokens: int64>
  child 0, sieve_dir: string
  child 1, mem_fraction: double
  child 2, max_batch_tokens: int64
model_source: struct<kind: string, path: string, device: string, dtype: string, base_model: string, base_revision: (... 228 chars omitted)
  child 0, kind: string
  child 1, path: string
  child 2, device: string
  child 3, dtype: string
  child 4, base_model: string
  child 5, base_revision: string
  child 6, list_options: bool
  child 7, built_in_temperature: double
  child 8, context_limit_tokens: int64
  child 9, adapter_model.safetensors_sha256: string
  child 10, head.safetensors_sha256: string
  child 11, sieve_engine.py_sha256: string
  child 12, modeling_sieve.py_sha256: string
torch: string
transformers: string
peft: string
flash-linear-attention: string
causal-conv1d: string
suite: struct<edition: string, selected_rows_sha256: string, added_rows_sha256: string, gsm8k_rows_sha256:  (... 48 chars omitted)
  child 0, edition: string
  child 1, selected_rows_sha256: string
  child 2, added_rows_sha256: string
  child 3, gsm8k_rows_sha256: string
  child 4, exclusions_sha256: string
  child 5, note: string
runner_sha256: string
latency: string
execution: struct<part_1: struct<rows: string, hardware: string, mode: string, started_utc: timestamp[s], finis (... 192 chars omitted)
  child 0, part_1: struct<rows: string, hardware: string, mode: string, started_utc: timestamp[s],
...
     child 1, score: null
              child 2, scored_requests: int64
          child 3, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
              child 0, metric: string
              child 1, score: null
              child 2, scored_requests: int64
          child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
              child 0, metric: string
              child 1, score: double
              child 2, scored_requests: int64
          child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
              child 0, metric: string
              child 1, score: double
              child 2, scored_requests: int64
edition: string
note: string
counts: struct<ok: int64, unsupported: int64>
  child 0, ok: int64
  child 1, unsupported: int64
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
                  raise CastError(
                  ...<3 lines>...
                  )
              datasets.table.CastError: Couldn't cast
              engine: string
              engine_options: struct<sieve_dir: string, mem_fraction: double, max_batch_tokens: int64>
                child 0, sieve_dir: string
                child 1, mem_fraction: double
                child 2, max_batch_tokens: int64
              model_source: struct<kind: string, path: string, device: string, dtype: string, base_model: string, base_revision: (... 228 chars omitted)
                child 0, kind: string
                child 1, path: string
                child 2, device: string
                child 3, dtype: string
                child 4, base_model: string
                child 5, base_revision: string
                child 6, list_options: bool
                child 7, built_in_temperature: double
                child 8, context_limit_tokens: int64
                child 9, adapter_model.safetensors_sha256: string
                child 10, head.safetensors_sha256: string
                child 11, sieve_engine.py_sha256: string
                child 12, modeling_sieve.py_sha256: string
              torch: string
              transformers: string
              peft: string
              flash-linear-attention: string
              causal-conv1d: string
              suite: struct<edition: string, selected_rows_sha256: string, added_rows_sha256: string, gsm8k_rows_sha256:  (... 48 chars omitted)
                child 0, edition: string
                child 1, selected_rows_sha256: string
                child 2, added_rows_sha256: string
                child 3, gsm8k_rows_sha256: string
                child 4, exclusions_sha256: string
                child 5, note: string
              runner_sha256: string
              latency: string
              execution: struct<part_1: struct<rows: string, hardware: string, mode: string, started_utc: timestamp[s], finis (... 192 chars omitted)
                child 0, part_1: struct<rows: string, hardware: string, mode: string, started_utc: timestamp[s],
              ...
                   child 1, score: null
                            child 2, scored_requests: int64
                        child 3, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: null
                            child 2, scored_requests: int64
                        child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: double
                            child 2, scored_requests: int64
                        child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
                            child 0, metric: string
                            child 1, score: double
                            child 2, scored_requests: int64
              edition: string
              note: string
              counts: struct<ok: int64, unsupported: int64>
                child 0, ok: int64
                child 1, unsupported: int64
              to
              {'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
              ...
              Value('int64'), 'mean': Value('int64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
              because column names don't match

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

Sieve-2B-v2: Decision Index results

The complete run of Sieve-2B-v2 on the public suite of the Decision Index 0.3, made with the official kit apolinario/decision-index at 62d2f51 (edition 0.3); the kit's own scorer produced the scores.

Leaderboard Full score 30.84, public score 37.26 (edition 0.3.2, board data of 2026-10-12); no longer listed since edition 0.4
This run Decision Index 0.3: public index 37.26 (raw 52.73, breadth 34.02)
Requests 140,178: 140,176 answered, 2 unsupported (options with identical text), 0 errors

In edition 0.3.2 the Full score counted the public benchmarks 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

area skill (×100)
Knowledge & Reasoning 16.6
Language Understanding 35.9
Retrieval & Classification 57.4
Tools & Automation 56.6
Arts & Human Taste 18.5

Files (runs/Sieve-2B-Plus/)

The run was made when the model was named Sieve-2B-Plus, so the run folder and the files in it use that name.

file contents
scores.json the kit's scores: "complete": true, edition 0.3, index, areas, per-benchmark values
index.json, benchmark-summary.json per-benchmark raw, skill and coverage; native metrics and counts
environment.json, status.json engine, options, package hashes, versions, suite hashes, and how the run was executed
results.jsonl.gz one result per request: run id, status, the model's answers with a probability for every option, timings

results.jsonl.gz is in the runner's --compact form: payload (benchmark text) and raw_output are left out, because several suite sources do not allow their text to be republished. Every row keeps its payload_sha256. Scoring this file with python -m decision_index score --edition 0.3 --results runs/Sieve-2B-Plus/results.jsonl.gz reproduces scores.json.

How the run was made

  • Suite. Rebuilt with suite rebuild; the 0.2.1 rows, the rebuilt GSM8K rows and the exclusions match the kit's pinned hashes, and suite verify --edition 0.3 passes. Raw suite inputs are not redistributed.
  • Model. sthanika-ai/Sieve-2B-v2, a LoRA adapter (r 64) and a scalar head on Qwen/Qwen3.5-2B-Base at b1485b2f, bf16, built-in temperature 1.4785, context limit 32,768 tokens. Engine sieve_engine:SieveEngine, then shipped in the model repo and now sieve.fork.engine.ForkEngine in github.com/sthanika-ai/Sieve (run it with --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-2B-v2), options max_batch_tokens=49152 and mem_fraction=0.16. Nothing is truncated, no option is dropped, no prompt is changed per benchmark.
  • Execution, in two parts with the same engine, options, package, runner and pinned versions:
    1. The 0.2.1 suite on 2× A100 80GB PCIe: 24 round-robin shards of exactly the rows pipeline runs, 10 processes, merged by run id with no duplicates and no retries.
    2. The 2,638 rebuilt GSM8K requests of 0.3 on one A100 80GB PCIe (RunPod), appended to that file exactly as resuming it under 0.3 would. A parity check of 50 requests on both machines gave a median probability difference of 1.2e-5 and one near-tie decision difference (bf16 kernels are not bit-identical across machines).
  • Latency (single process, one request at a time, bf16, A100 80GB, 400 uniform suite requests): median 97 ms, mean 124 ms, 80th percentile 110 ms. The latency_ms in scores.json was recorded in the shared 10-process run.
Downloads last month
194
Total file size:
44.2 MB

Collection including sthanika-ai/Sieve-2B-v2-decision-index-results