Repository navigation
eval: report saturated, always-fail and inverted scenarios with per-category headroom #4477
Description
Nothing tells us which scenarios have stopped measuring anything. Three cases hide in the pass rate:
- A scenario every model passes has no headroom left.
- One the strongest model always fails is usually an ambiguous task or a rubric that disagrees with the task.
- One where a weaker model beats a stronger one is measuring noise.
All three inflate or deflate the pass rate without saying anything about the agent. As ~128 new tier scenarios land, this gets harder to spot by eye.
Next step: add a scenario health report over recent nightly scorecards (and optional strong/weak model runs) that flags saturated, always-fail, inverted and flaky scenarios, plus a headroom number per category.
Part of the eval design & hillclimbing plan (docs/plans/eval-hillclimbing.mdx, item 5).
🔍 Technical details
Proposed
gaia eval agent health RUN_DIR... [--strong RUN_DIR] [--weak RUN_DIR], in a newsrc/gaia/eval/health.py. It readsscorecard.jsonfiles and uses the per-scenariostabilityblock where present.- Saturated:
stable-passon every model and run. - Always-fail:
stable-failon the strongest model. - Inverted: the weak model's mean score is above the strong model's.
- Flaky: the pass rate is strictly between 0 and 1.
- Saturated:
- Per category, report headroom: the strongest model's judged pass rate. Warn above 95%.
- Output
health.jsonplus a markdown table for the job summary. - A weekly job in
eval_flagship.ymlpublishes the report. Always-fail and inverted scenarios get routed to a human (task or rubric fix). Saturated ones become candidates for a cheaper smoke lane.
Acceptance
- Unit tests cover each flag using synthetic scorecards.
- One real report is attached to the PR.
docs/reference/eval.mdxdocuments it.
Activity
docs(plans): eval design & hillclimbing plan (amd#4478)