S1-Arena

S1-Arena asks which decisions a fast System-1 model can make, and which still need a generative LLM. Every request holds an input (state) and one or more typed questions: choice (one of 2 to 255 options), boolean (the probability that a statement is true) or score (2 to 10 ordered levels). A model answers each question with a probability distribution over its options.

This dataset holds 49,592 built requests from 254 sources in 16 domains (one config per domain, split test). Code: https://github.com/TianyuCodings/S1-Arena · leaderboard: https://tianyucodings.github.io/S1-Arena.

Use

git clone https://github.com/TianyuCodings/S1-Arena && cd S1-Arena && uv sync
uv run s1arena download        # writes data/<task>.jsonl
uv run s1arena build --local   # rebuilds the sources that do not allow redistribution

from datasets import load_dataset
rows = load_dataset("C-Tianyu/S1-Arena", "basic_cls", split="test")

Columns

Domains

Closed-loop environments

environments.json lists the 138 closed-loop environments (games, web and text agents, embodied control, realtime control) with their seeds. They run in simulators, not from this dataset: s1arena play.

Licenses

Each source keeps its own license; check it before you use a source. The code of S1-Arena is Apache-2.0. Requests that S1-Arena generates are licensed CC BY 4.0.

Not included (14 sources)

These sources do not allow redistribution. s1arena build --local rebuilds their requests from the original data (people invited to the private dataset C-Tianyu/S1-Arena-restricted get them with s1arena download).

Citation

@article{s1arena2026,
  title  = {System1-Arena: Which Decisions Can Be Left to System 1?},
  author = {Chen, Tianyu and Liu, Xinyang and An, Ruyi and Wang, Zhendong and Zhou, Mingyuan},
  journal = {arXiv preprint arXiv:2610.21244},
  year   = {2026}
}