Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
engine: string
engine_options: struct<repo: string, calibrate: bool, batch_size: int64>
child 0, repo: string
child 1, calibrate: bool
child 2, batch_size: int64
model_source: struct<repo: string, revision: string, backend: string, device: string, dtype: string, calibrated: b (... 72 chars omitted)
child 0, repo: string
child 1, revision: string
child 2, backend: string
child 3, device: string
child 4, dtype: string
child 5, calibrated: bool
child 6, max_length: int64
child 7, experts: list<item: string>
child 0, item: string
child 8, truncation: string
loaded_seconds: double
frozen_corpus_sha256: string
rows_path: list<item: string>
child 0, item: string
runner_sha256: string
latency: string
counts: struct<ok: int64, unsupported: int64>
child 0, ok: int64
child 1, unsupported: int64
edition: string
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6266 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6254 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: double
child 11, referen
...
hild 2, scored_requests: int64
child 3, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
note: string
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
lue('int64'), 'mean': Value('float64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
engine: string
engine_options: struct<repo: string, calibrate: bool, batch_size: int64>
child 0, repo: string
child 1, calibrate: bool
child 2, batch_size: int64
model_source: struct<repo: string, revision: string, backend: string, device: string, dtype: string, calibrated: b (... 72 chars omitted)
child 0, repo: string
child 1, revision: string
child 2, backend: string
child 3, device: string
child 4, dtype: string
child 5, calibrated: bool
child 6, max_length: int64
child 7, experts: list<item: string>
child 0, item: string
child 8, truncation: string
loaded_seconds: double
frozen_corpus_sha256: string
rows_path: list<item: string>
child 0, item: string
runner_sha256: string
latency: string
counts: struct<ok: int64, unsupported: int64>
child 0, ok: int64
child 1, unsupported: int64
edition: string
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6266 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6254 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: double
child 11, referen
...
hild 2, scored_requests: int64
child 3, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
note: string
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
lue('int64'), 'mean': Value('float64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Dinah-1 on the Decision Index 0.3
Complete run of Lukitaduarte/dinah-1 on the Decision Index suite, edition 0.3.
| Decision Index 0.3 (public index) | 34.32 (raw 51.36) |
| Requests | 140,620 run; 140,178 scoreable after the kit's exclusions, all with a result except 8 unsupported |
| Model | Lukitaduarte/dinah-1 @ b518a5e773597a00b13cfdae928ca1fd65c208f1 |
| Engine | decision_index_engine:Dinah01Engine (decision_index_engine.py, dinah.py, experts.py and tools.py in the model repository), code and calculator experts on, bfloat16, batch of 8 questions |
| Kit | apolinario/decision-index edition 0.3 (suite-0.3, runner sha256 f4c9f36c...) |
| Hardware | 1 x NVIDIA RTX PRO 5000. Two processes (calibrated and uncalibrated) ran at the same time on the same GPU, so the latency below is not a stand-alone figure |
| Latency (this run, shared GPU) | median 20.8 ms, p95 76.4 ms, mean 41.3 ms per request |
| Declared limits | 8,192-token context. Nothing is truncated: the 324 requests that do not fit are refused as unsupported (8 of them scoreable). |
Two runs are included. runs/dinah-1/ is the engine default, with the calibration shipped in the model's config.json. runs/dinah-1-uncalibrated/ ran the same weights with calibrate=false. The calibration changes probabilities and confidence but no chosen option, and the index is 34.32 in both runs.
Both runs used a local copy of the files that were later published in the revision above. The weights (model.safetensors, calculator/model.safetensors), config.json, the tokenizer files, tools.py and decision_index_engine.py are byte-identical to the published ones. dinah.py and experts.py changed afterwards only in their docstrings and in the default repository id (Lukitaduarte/dinah-1 instead of Lukitaduarte/dinah-0-apache). environment.json shows the local path of the copy that was run. SHA-256 of the files that were run:
3322bfc29d3775ce46b1beac074baaa298bcdc14f131d35efc5fd827ba320dd3 model.safetensors
49d4c96dd30d378fe295b865306111928494ee48ae72530f191daa0aaf6f1629 calculator/model.safetensors
3be30f3f7ac3de1bb7f9927704aee5cdeb20db0c439e1e9429fd7306da118fe6 config.json
7fb3a94d494d86f35ee697a6fe7b55191b06ece7929f63bcb8331543a3aeceba decision_index_engine.py
2a817b3e9f1a0c2c2e5a9566c88ec225229076dd5b04ff2387e8ac8800c2e7a7 dinah.py
ac0e4905ef64d39d49388534312f38ff5b507cc5194503b70a110b6fadda96e9 experts.py
59895a0d6a79540c17008dcca39b496b67317a4a8a7f8462d480663427cd6c04 tools.py
edeac9e7b2f170022cefd1c7b396ebca3ac9c57f2f93604844cb7fb16b1a71ca tokenizer.json
278c882670a00adc366a42c1c698934a60dc12ea9dd14e5eaa7198a81d761c10 tokenizer_config.json
519ed82150132a0f7822de540df23c0803ad718f624e61e6a90113867f9b5a2f special_tokens_map.json
The latency of the experts: the calculator expert runs a beam search of up to 20 encoder passes per problem (about 0.6 s on a GSM8K problem on this GPU) and the code expert about 33 ms per CRUXEval request, against about 20 ms for a request without an expert. The median is therefore close to the encoder's, while the mean and the tail include the experts.
Each run directory holds the runner output (results.jsonl, written with --compact, so no suite inputs are redistributed), scores.json, index.json, benchmark-summary.json, environment.json and status.json.
- Downloads last month
- 70
