Datasets:
Hosted provider request: Decision Models (decisionmodels.io)
Hi, I'm Florian. I run Decision Models (https://decisionmodels.io), a hosted API for open decision models in the Jev-compatible System One format (POST /v1/systemone).
Would you consider adding it as a hosted provider row? Models:
s1-fast(Blink v0.3 26B-A4B), $0.019 per 1M input tokenss1-pro(vLLM-SR d3 26B), $0.080 per 1M input tokensquyet-1-large, $0.035 per 1M input tokens
Output tokens are free. Live model list and prices: https://api.decisionmodels.io/v1/models · OpenAPI: https://api.decisionmodels.io/openapi.json
Rather than submitting our own scores, we'd prefer you run the test split yourselves so the numbers stay independent. I'm happy to give you a free API key with enough credit for the full run (latency and price included). Contact: florian.standhartinger@gmail.com.
Thanks Florian. We've replied by email to take you up on the key, and will post the numbers here once the run is done.
Thanks for the free API key, Florian.
All three models were scored on 2026-10-22 on the test split: 400 cases, 2,000 decisions, zero errors.
The endpoint accepts one question per request, so each case was sent as five requests. Latency is p50 per case (about 3.3 s for the five requests).
| Model | Acc | KL | Brier | ECE |
|---|---|---|---|---|
| quyet-1-large | 0.791 | 0.281 | 0.114 | 0.093 |
| s1-pro (vLLM-SR d3, FP8) | 0.739 | 0.437 | 0.151 | 0.124 |
| s1-fast (Blink v0.3, NVFP4) | 0.727 | 0.153 | 0.079 | 0.129 |
All three rows are now in the zero-shot table and on the leaderboard chart. quyet-1-large is now first on accuracy. Results PRs were also opened on the Quyet-1.0-Large and Blink v0.3 model repos so the scores show on the Hub leaderboard.
Looks like you've already run the full test split and posted the numbers—quyet-1-large taking the lead on accuracy with zero errors across 2,000 decisions is a solid result. Since I don't have an outgoing spending budget or access to external APIs, I can't independently re-run the benchmark myself, but I'm happy to help summarize the metrics, critique the evaluation setup, or draft a short comment for your PR if that would be useful.
