refactor!: remove NematodeBench, preserving the convergence detector and session data - #274
Merged
Conversation
NematodeBench was demoted from a Phase 7 deliverable to "internal tooling"
at the v4 roadmap rewrite, justified by usefulness to the
architecture-comparison protocol. Two phases of evidence say otherwise: the
protocol read --track-experiment output directly via
weight_search_architecture_ranking.py and never invoked the submission
pipeline. The corpus is six submissions from one contributor in Dec 2025,
covering 3 of 27 brain architectures; BENCHMARKS.md still advertises a
static_maze category deleted from the code in Jan 2026.
This is a removal-plus-migration, not a straight deletion. Three
requirements in benchmark-management specify live behaviour and move to
experiment-tracking: Reproducibility Through Seeding (the only live-spec
coverage of single-agent seeding, which the paired-seed statistics rest
on), the two convergence scenarios of Enhanced Metrics for Benchmarks, and
the experiment-folder storage scenarios. The migration also surfaced a
drift â experiment-tracking specifies a flat experiments/{id}.json while
the code writes the folder form â corrected here.
convergence.py is retained and relocated to experiment/; it is the ranked
metric producer for architecture-comparison-protocol, not submission code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found one blocking issue and two gaps. Blocking: the 72 experiment JSONs under artifacts/benchmarks/ have no duplicate â their IDs are disjoint from the 12 under artifacts/experiments/. Deleting the directory would destroy the sole copy of 72 tracked experiment records with their per-run seeds. They are primary data, not submission tooling, so section 4 now migrates them into the live artifacts/experiments/<id>/ layout and removes only the six submission manifests (design.md D6). Gaps: the final-grep acceptance criterion missed a fourth surviving hit (Logbook 009's orphaned "Formal NematodeBench submission" checkbox, now handled in place per the phase6-tracking stub precedent); and task 2.5 could have swept run_simulation.py:1205, which refers to ChemotaxisValidationBenchmark â the real-worm validation, a different system â so it is now explicitly fenced off. Also records the local verification baseline: this laptop cannot run the pinned numpy/torch, so pytest is 3-failed and pyright 61-errors before the change starts. The bar is that neither count increases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deletes the submission/validation/leaderboard layers, which had no runtime
consumer: benchmark/{leaderboard,categorization,validation}.py,
experiment/{submission,validation}.py, BenchmarkMetadata, and the two CLI
entry points (benchmark_submit.py, evaluate_submission.py). Drops the
orphaned ruff per-file-ignores and the codecov exclusion they justified.
BREAKING: quantumnematode.experiment no longer exports
NematodeBenchSubmission, SessionReference, AggregateMetrics,
BenchmarkMetadata, validate_submission, MIN_SESSIONS_REQUIRED or
MIN_RUNS_PER_SESSION.
Adds test_from_dict_ignores_removed_benchmark_key as the silent-break
guard. 421 tracked artifacts carry a top-level "benchmark" key and they
keep loading only because Pydantic's default extra="ignore" drops it â
inherited behaviour, not declared. Adding extra="forbid" to
ExperimentMetadata would turn every one of those reads into a
ValidationError with nothing to catch it. The test also pins
composite_benchmark_score, which is deliberately not renamed: a rename
would read as None across all 421 artifacts without raising.
Leaves run_simulation.py:1205 untouched â "benchmark" there refers to
ChemotaxisValidationBenchmark, the real-worm biological validation.
Verified: pytest 3 failed / 4059 passed, pyright 61 errors â both
byte-identical to the recorded pre-change baseline (this laptop cannot run
the pinned numpy/torch). ruff, mdformat and markdownlint clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
benchmark/convergence.py is the ranked-metric producer for architecture-comparison-protocol, not submission code â it got feature work as recently as 2026-06-21 and every --track-experiment run calls it. With the submission layers gone, the benchmark/ package held only this module, so it moves to its consumer. experiment/ over a new analysis/ package: scripts/analysis/ and tests/quantumnematode_tests/analysis/ already own that name (the latter path-injects and tests the former), and metrics/ is taken twice (agent/metrics.py, plasticity/metrics.py). experiment/ is where the sole caller and the ResultsMetadata model it populates already live, and the move adds no new import edge â tracker.py already imports report.dtypes, convergence.py's only non-stdlib dependency. No symbol renames. Updates the live path reference in architecture-comparison-protocol; leaves the identical paths under openspec/changes/archive/ alone, as those record the path as it was. Verified: 22 convergence tests pass at the new path, 95 experiment tests green, pyright byte-identical to the 61-error baseline, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 72 experiment JSONs under artifacts/benchmarks/ had no duplicate anywhere â their IDs and the 12 under artifacts/experiments/ were disjoint sets, so deleting the directory would have destroyed the only copy of 72 tracked experiment records with their per-run seeds. They are primary experiment data, not submission tooling. Migrated into the live self-contained folder layout, artifacts/experiments/<id>/<id>.json plus the originating config named after the experiment's own config_file field (ppo_foraging_small.yml etc.) rather than the generic config.yml the submission grouping used. Moved with git mv so the LFS objects are renamed, not re-added: all 72 stage as 3-line pointers and git lfs status reports unchanged OIDs. Deletes only the six submission manifests under benchmarks/, which were aggregate roll-ups produced by the now-removed pipeline and meaningless without it. Drops the orphaned benchmarks/**/*.json LFS rule and the !benchmarks/ gitignore negation; the generic artifacts/**/*.json rule already covers the migrated files, verified via git check-attr. Verified: artifacts/experiments/ is 12 -> 84 entries, no ID collisions, all 84 JSONs parse and validate as ExperimentMetadata, and every migrated folder holds exactly one JSON and one config. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deletes BENCHMARKS.md (no content commit since 2025-12-28; still advertised a static_maze category removed from the code in Jan 2026) and the five files under docs/nematodebench/. Strips the generated leaderboard section from README.md, the submission / quality-standards / workflow / leaderboard subsections from CONTRIBUTING.md along with the benchmarks/ artifact-table row and the scripts bullet, the promote-to-benchmark step from the experiments workflow diagram, and the three AGENTS.md entries. CONTRIBUTING's "Experiment Tracking and Benchmarks" section is retitled to "Experiment Tracking" â the tracking and query content it also held is live and stays. Deliberately untouched: CONTRIBUTING's Nightly E2E Tests section and its e2e_benchmarks.json reference, which is the regression-ceiling system, not NematodeBench. Logbook 009's orphaned "Formal NematodeBench submission for lstmppo configs" action item is struck in place rather than deleted. Logbooks record what was true, but an open checkbox is a forward commitment, and that one now points at a system that does not exist. Verified: no dead links remain in user-facing docs. The surviving NematodeBench mentions are the roadmap and ADR (rewritten next), the live specs (edited at archival), the change document itself, and the struck logbook line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The v4 roadmap demoted NematodeBench from a Phase 7 public-launch
deliverable to internal tooling, on the stated grounds that it stayed
useful to the architecture-comparison protocol. Removing it reverses a
recorded decision, so the reversal is written down with its evidence rather
than left implicit in a deletion diff.
Roadmap: the Future Directions section is retitled "(removed 2026-07-25)"
and states what went, what survives, and why the retention argument did not
hold. The public-launch section keeps its "benchmarks crystallise mature
communities" reasoning â still correct â and notes only that the fallback
it preserved is gone, so a future launch is a from-scratch build. The v3
reversal table gains a new row directly under the original demotion row;
that table records what changed when, so history is appended, not edited.
Principle 4 ("demote rather than delete") gains a corollary: demotion is a
holding position, and where a demoted component then accrues no use across
a full phase, deletion follows and is recorded.
STANDARDIZATION ADR: superseded in place. The original Decision and
Rationale stay verbatim; a "Why superseded" block explains that the premise
(no external framework fits our domain metrics) was an argument for
domain-specific metric capture, not for a submission pipeline, and that the
decision conflated them. Notes that the two Phase-2 enhancements which
mattered â hierarchical categories and statistical testing â were delivered
inside architecture-comparison-protocol instead, as paired-seed Wilcoxon +
bootstrap + BH-FDR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applies the four spec deltas and archives the change.
experiment-tracking gains the three requirements that were living in
benchmark-management despite never being benchmark behaviour:
Reproducibility Through Seeding (verbatim â the only live-spec coverage of
single-agent seeding, which the paired-seed statistics rest on),
Convergence-Derived Metrics (the two live scenarios of the former Enhanced
Metrics for Benchmarks), and the experiment-folder storage scenarios.
Corrects the storage drift the migration surfaced: two scenarios specified
a flat experiments/{id}.json while storage.py writes the folder form
experiments/<id>/<id>.json and discovers experiments by scanning
subdirectories. Both are now accurate.
cli-interface loses the Benchmark Management CLI requirement and the
--save-benchmark / --benchmark-notes scenarios; those flags were never
implemented, so that part is drift correction rather than removal.
environment-simulation loses Predator-Enabled Benchmark Categories and the
Benchmark Category Name Verification scenario, both of which specified the
deleted categorization module.
benchmark-management is tombstoned rather than deleted â no capability spec
has ever been deleted here, and a reader arriving from an archived change
needs to find where the live parts went, not a 404. The tombstone carries
the migration table and names the three unrelated systems that also contain
the word "benchmark", so a future grep does not mistake them.
Also fixes a second copy of the stale "added for NematodeBench format"
comment in tracker.py that the first pass missed.
Verified end to end: a 20-run --track-experiment simulation still writes
composite_benchmark_score (0.833), learning_speed, stability and the
post-convergence fields, with no benchmark key in the output. pyright
byte-identical to the 61-error baseline; pytest 2 failed / 4060 passed
against a 3-failure baseline (the third, test_cry_crz_deterministic, is
intermittent and passed this run); ruff, mdformat and markdownlint clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedToo many files! This PR contains 174 files, which is 24 over the limit of 150. To get a review, narrow the scope: Upgrade to Pro+ to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. âïļ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ð Files selected for processing (199)
You can disable this status message by setting the âĻ Finishing Touches𧊠Generate unit tests (beta)
Comment |
Codecov Reportâ All modified and coverable lines are covered by tests. ðĒ Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of several pre-Phase-7 cleanup PRs. Removes NematodeBench â the curated benchmark submission, validation and leaderboard system â while preserving the one piece of live code inside it and all of its experimental data.
OpenSpec change:
2026-07-25-remove-nematodebench.Why
The v4 roadmap rewrite (2026-05-23) demoted NematodeBench from a Phase 7 public-launch deliverable to internal tooling, justified by its usefulness "for reproducibility and for the architecture-comparison protocol itself". Two phases of evidence contradict that:
--track-experimentoutput directly viascripts/analysis/weight_search_architecture_ranking.pyand never invoked the submission pipeline across Phases 5 or 6 â no logbook uses it.BENCHMARKS.mdhad no content commit for 19 months and still advertised astatic_mazecategory deleted from the code in9a452fd5(2026-01-30).That is ~2.4k lines of unmaintained, un-exercised surface being carried into Phase 7 for no consumer.
What survives
convergence.pyis not submission code â it is the ranked-metric producer forarchitecture-comparison-protocol, it received feature work as recently as 2026-06-21, and every--track-experimentrun calls it. Moved toexperiment/convergence.py, co-located with its only caller and theResultsMetadatamodel it populates. No symbol renames.All 72 session experiments are preserved. Their IDs and the 12 already under
artifacts/experiments/are disjoint sets, so deletingartifacts/benchmarks/would have destroyed the sole copy of 72 tracked experiment records with their per-run seeds. They are migrated into the live folder layout instead â each with its originating config (ppo_foraging_small.ymletc.) rather than the genericconfig.ymlthe submission grouping used. Moved viagit mv, so LFS OIDs are unchanged. Only the six aggregate manifests are deleted.Three unrelated systems that share the word "benchmark" are untouched:
scripts/benchmarks/bench_evolution_smoke.py(evolution perf harness),ChemotaxisValidationBenchmark(real-worm biological validation), andtests/.../e2e_benchmarks.json(nightly regression ceilings).Spec work: a removal and a migration
Auditing all 17
benchmark-managementrequirements individually â rather than assuming removability from the capability name â found three that specify genuinely live behaviour. These migrate toexperiment-tracking:Reproducibility Through Seedingmulti-agentcovers only the multi-agent case). The paired-seed statistics the whole research programme rests on depend on it.Enhanced Metrics for Benchmarkscalculate_learning_speed/calculate_stability, which survive.Experiment Storage and Trackingexperiments/andartifacts/experiments/, both live.The migration also surfaced a spec/code drift:
experiment-trackingspecified a flatexperiments/{id}.jsonin two places, whilestorage.pywrites the folder formexperiments/<id>/<id>.jsonand discovers experiments by scanning subdirectories. Both are corrected.benchmark-managementis tombstoned rather than deleted â no capability spec has ever been deleted in this repo, and a reader arriving from an archived change needs to find where the live parts went, not a 404. The tombstone carries the migration table and names the three confusable systems.Notable decision:
composite_benchmark_scorekeeps its nameTempting to rename now that "benchmark" no longer names a subsystem. 421 tracked artifacts on disk carry the key, and
ExperimentMetadata.from_dictends incls(**data)with Pydantic's defaultextra="ignore"â so a rename would populate the new field from a missing key and returnNone, silently, with no exception.report/csv_export.pyalso emits the string as a literal CSV label. Only two stale comments changed.That same inherited-not-declared behaviour is what keeps historical artifacts loading at all (420 carry
"benchmark": null, one is populated), so this PR addstest_from_dict_ignores_removed_benchmark_keyas a guard â if anyone later addsextra="forbid"toExperimentMetadata, that test fails loudly instead of 421 reads breaking quietly.Verification
End-to-end
--track-experimentrun (20 runs,mlpppo_small_oracle) confirms the tracking pipeline survived the move:composite_benchmark_score0.833, pluslearning_speed,stabilityand the post-convergence fields all written, and nobenchmarkkey in new output.numpy>=2.2.4/torch>=2.7.0and has numpy 1.26.4 / torch 2.2.0 installed, producing float32-vs-float64 boundary failures (e.g. float32-1.5707964failing a>= -Ï/2bound by ~1e-7) and numpy-1.x scalar typing errors. The bar applied here was that neither count increases, and neither did. CI runs the pinned versions and is the authority.Also worth flagging independently of this PR:
test_qef.py::TestQEFGateAndFeatureModes::test_cry_crz_deterministicproved intermittent â it failed one local run and passed the next with no code change between them.Migration notes
Breaking:
quantumnematode.benchmarkis gone as an import path, andquantumnematode.experimentno longer exportsNematodeBenchSubmission,SessionReference,AggregateMetrics,BenchmarkMetadata,validate_submission,MIN_SESSIONS_REQUIREDorMIN_RUNS_PER_SESSION.scripts/benchmark_submit.pyandscripts/evaluate_submission.pyare deleted. No external API consumers, so no deprecation window.The reversal is recorded in
docs/roadmap.md(new row appended to the v3 reversal table rather than editing the existing one) anddocs/STANDARDIZATION.md(ADR superseded in place, original rationale kept verbatim). Roadmap principle 4, "demote rather than delete", gains a corollary: demotion is a holding position, and where a demoted component then accrues no use across a full phase, deletion follows and is recorded.ðĪ Generated with Claude Code