Dataset Viewer
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code: StreamingRowsError
Exception: CastError
Message: Couldn't cast
engine: string
engine_options: struct<repo: string, calibrate: bool, batch_size: int64>
child 0, repo: string
child 1, calibrate: bool
child 2, batch_size: int64
model_source: struct<repo: string, revision: string, device: string, dtype: string, max_length: int64, calibrated: (... 26 chars omitted)
child 0, repo: string
child 1, revision: string
child 2, device: string
child 3, dtype: string
child 4, max_length: int64
child 5, calibrated: bool
child 6, truncation: string
loaded_seconds: double
frozen_corpus_sha256: string
rows_path: list<item: string>
child 0, item: string
runner_sha256: string
latency: string
note: string
counts: struct<ok: int64, unsupported: int64>
child 0, ok: int64
child 1, unsupported: int64
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6267 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6255 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: double
child 11, reference_same_cases: null
child 12, median_ms: double
child 13, detail: struct<field_accura
...
d 2, scored_requests: int64
child 3, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
edition: string
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
lue('int64'), 'mean': Value('float64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't match
Traceback: Traceback (most recent call last):
File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
return get_rows(
dataset=dataset,
...<4 lines>...
column_names=column_names,
)
File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
return func(*args, **kwargs)
File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
yield from ds.decode(False) if ds.features else ds
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
for key, example in ex_iterable:
^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
for key, pa_table in self._iter_arrow():
~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
for key, pa_table in self.ex_iterable._iter_arrow():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
for key, pa_table in iterator:
^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
for key, pa_table in self.generate_tables_fn(**gen_kwags):
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
self._cast_table(pa_table, json_field_paths=json_field_paths),
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
pa_table = table_cast(pa_table, self.info.features.arrow_schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
return cast_table_to_schema(table, schema)
File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
raise CastError(
...<3 lines>...
)
datasets.table.CastError: Couldn't cast
engine: string
engine_options: struct<repo: string, calibrate: bool, batch_size: int64>
child 0, repo: string
child 1, calibrate: bool
child 2, batch_size: int64
model_source: struct<repo: string, revision: string, device: string, dtype: string, max_length: int64, calibrated: (... 26 chars omitted)
child 0, repo: string
child 1, revision: string
child 2, device: string
child 3, dtype: string
child 4, max_length: int64
child 5, calibrated: bool
child 6, truncation: string
loaded_seconds: double
frozen_corpus_sha256: string
rows_path: list<item: string>
child 0, item: string
runner_sha256: string
latency: string
note: string
counts: struct<ok: int64, unsupported: int64>
child 0, ok: int64
child 1, unsupported: int64
benchmarks: list<item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: (... 6267 chars omitted)
child 0, item: struct<catalog_id: int64, dataset: string, requests: int64, answered: int64, unsupported: int64, err (... 6255 chars omitted)
child 0, catalog_id: int64
child 1, dataset: string
child 2, requests: int64
child 3, answered: int64
child 4, unsupported: int64
child 5, errors: int64
child 6, abstained: int64
child 7, pending: int64
child 8, scored_requests: int64
child 9, metric: string
child 10, score: double
child 11, reference_same_cases: null
child 12, median_ms: double
child 13, detail: struct<field_accura
...
d 2, scored_requests: int64
child 3, iSarcasmEval-A-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 4, iSarcasmEval-B-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 5, iSarcasmEval-C-Ar: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 6, iSarcasmEval-C-En: struct<metric: string, score: null, scored_requests: int64>
child 0, metric: string
child 1, score: null
child 2, scored_requests: int64
child 7, GSM8K-10choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
child 8, GSM8K-4choice: struct<metric: string, score: double, scored_requests: int64>
child 0, metric: string
child 1, score: double
child 2, scored_requests: int64
successful_request_latency_ms: struct<median: double, p95: double, mean: double>
child 0, median: double
child 1, p95: double
child 2, mean: double
edition: string
to
{'engine': Value('string'), 'counts': {'ok': Value('int64'), 'unsupported': Value('int64')}, 'successful_request_latency_ms': {'median': Value('float64'), 'p95': Value('float64'), 'mean': Value('float64')}, 'benchmarks': List({'catalog_id': Value('int64'), 'dataset': Value('string'), 'requests': Value('int64'), 'answered': Value('int64'), 'unsupported': Value('int64'), 'errors': Value('int64'), 'abstained': Value('int64'), 'pending': Value('int64'), 'scored_requests': Value('int64'), 'metric': Value('string'), 'score': Value('float64'), 'reference_same_cases': Value('null'), 'median_ms': Value('float64'), 'detail': {'field_accuracy': Value('float64'), 'case_exact_accuracy': Value('float64'), 'custom_metrics': {'ndcg_at_10': Value('float64'), 'mrr': Value('float64'), 'recall_at_10': Value('float64'), 'candidate_recall': Value('float64'), 'scorable_candidate_recall': Value('float64'), 'candidates_scored': Value('float64'), 'candidates_retrieved': Value('int64'), 'bm25_ndcg_at_10': Value('float64'), 'quality_quality': Value('float64'), 'quality_cost_usd': Value('float64'), 'quality_utility': Value('float64'), 'quality_oracle_optimal': Value('float64'), 'quality_utility_regret': Value('float64'), 'cost_aware_quality': Value('float64'), 'cost_aware_cost_usd': Value('float64'), 'cost_aware_utility': Value('float64'), 'cost_aware_oracle_optimal': Value('float64'), 'cost_aware_utility_regret': Value('float64'), 'value_regret': Value('float64'), 'normalized_rank_regret': Value('float6
...
lue('int64'), 'mean': Value('float64')}}, 'cluster_macro_accuracy': Value('float64'), 'positive_f1_by_field': {'sarcastic': Value('float64'), 'sarcasm': Value('float64'), 'irony': Value('float64'), 'satire': Value('float64'), 'understatement': Value('float64'), 'overstatement': Value('float64'), 'rhetorical_question': Value('float64')}, 'category_macro_f1': Value('float64'), 'scored_fields': Value('int64'), 'chance_on_rows': Value('float64')}, 'tracks': {'RouterBench-0shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'RouterBench-5shot': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-A-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-B-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-Ar': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'iSarcasmEval-C-En': {'metric': Value('string'), 'score': Value('null'), 'scored_requests': Value('int64')}, 'GSM8K-10choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}, 'GSM8K-4choice': {'metric': Value('string'), 'score': Value('float64'), 'scored_requests': Value('int64')}}}), 'note': Value('string'), 'edition': Value('string')}
because column names don't matchNeed help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Dinah-0 on the Decision Index 0.2.1
Complete, untouched run of Lukitaduarte/dinah-0 on the Decision Index suite.
| Decision Index 0.2.1 | 27.63 (raw 44.78) |
| Requests | 150,759 run; 150,317 scoreable, all with a result (8 of them unsupported) |
| Model | Lukitaduarte/dinah-0 @ e6bfc332fd31b6625cc912441c92f4a11dcb1266 |
| Engine | decision_index_engine:DinahEngine (decision_index_engine.py and dinah.py in the model repository, same revision), bfloat16, batch of 8 questions |
| Kit | apolinario/decision-index @ 87d4650 (runner sha256 f4c9f36c...) |
| Hardware | 1 x NVIDIA RTX PRO 4000 Blackwell, one process, one request at a time |
| Latency (this run) | median 5.6 ms, p95 52.5 ms per request |
| Declared limits | 8,192-token context. Nothing is truncated: the 324 requests that do not fit are refused as unsupported (8 of them scoreable). |
runs/dinah-0/ holds the runner output (results.jsonl, written with --compact, so no suite inputs are redistributed), scores.json, index.json, benchmark-summary.json, environment.json and status.json.
Dinah-0 Apache on the Decision Index 0.3
Complete run of Lukitaduarte/dinah-0-apache on the Decision Index suite, edition 0.3.
| Decision Index 0.3 (public index) | 25.86 (raw 44.50) |
| Requests | 140,620 run; 140,178 scoreable after the kit's exclusions, all with a result except 8 unsupported |
| Model | Lukitaduarte/dinah-0-apache @ 3265ba1ed23763716da52ecaea4af09d1a6eb613 |
| Engine | decision_index_engine:DinahEngine (decision_index_engine.py and dinah.py in the model repository), bfloat16, batch of 8 questions |
| Kit | apolinario/decision-index @ 9eb2dbe, edition 0.3 (suite-0.3, runner sha256 f4c9f36c...) |
| Hardware | 1 x NVIDIA RTX 5090. Two processes (calibrated and uncalibrated) ran at the same time on the same GPU, so the latency below is not a stand-alone figure |
| Latency (this run, shared GPU) | median 10.2 ms, p95 44.3 ms per request |
| Declared limits | 8,192-token context. Nothing is truncated: the 324 requests that do not fit are refused as unsupported (8 of them scoreable). |
Two runs are included. runs/dinah-0-apache/ is the engine default, with the calibration shipped in the model's config.json. runs/dinah-0-apache-uncalibrated/ ran the same weights with calibrate=false. The calibration changes probabilities and confidence but no chosen option: of 314,940 answers, none differ in the chosen label, and the index is 25.86 in both runs.
Both runs used a local copy of the files that were later published in the revision above (same weights, same engine and dinah.py code); only the display name of the model in the code and the repository name changed afterwards, so environment.json shows the local path of the copy that was run. For comparison, Dinah-0 (CC BY-NC 4.0) is listed at 26.06 on the live Decision Index 0.3 board.
- Downloads last month
- 116
