docs(logbook): T7 continuous-substrate cross-architecture ranking (029) - #255
Merged
Conversation
âĶWIP) Plateau-tail (final-quarter) success + foods as the robust ranked metric (reads the real convergence_run flag; avoids the last-10 fallback), reusing the committed paired-Wilcoxon/bootstrap/BH-FDR stats layer. Results + logbook land once the n>=8 @6000ep runs complete. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publish logbook 029 â the T7 (high-fidelity continuous-2D) analogue of 025: the
6-MUST-arm integrated-C3 ranking (n=8 paired seeds, plateau-tail full-clear @
uniform 6000ep + a 5-seed 8000ep convergence top-up; paired Wilcoxon + 80%
bootstrap + BH-FDR via the committed stats layer).
Ranking: MLP 89.0 >> {CfC 75.8 ~ Transformer 74.0} > LSTM 60.1 > connectome 52.2
>> GA 15.0 â three significant tiers, robust across budgets. MLP best on all
three behaviours; connectome 5th (T4 evasion-lag confirmed, but learns the cell
8/8 â not a STOP, G3.b); GA gradient-free floor; the sub-saturation cell
discriminates where T4's flat cluster tied.
- Main logbook + supporting/029 (per-seed CSV, pairwise-stats JSON, GA results,
budget-iteration + dig-in detail). Following 025's lighter artifact practice:
analysis outputs only (~28K), no raw session dumps or weights.
- Tracker: check off the 6 c3_integrated cells + analysis.{ranking_upgraded,
env_delta,connectome_final_ranking}. T7.logbook left UNCHECKED â T7 not closed
(SHOULD/MAY arms + a memory-bound control deferred).
- MLP config: remove the dead entropy_coef_end/entropy_decay_episodes keys
(mlpppo never implemented the schedule; it ran flat 0.08) + relabel the recipe.
- README active-experiments index updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. ð âđïļ Recent review infoâïļ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ð Files selected for processing (4)
â Files skipped from review due to trivial changes (2)
ð§ Files skipped from review as they are similar to previous changes (2)
ð WalkthroughWalkthroughAdds T7 ranking and GA evaluation scripts, supporting result data, the Experiment 029 logbook and appendix, removes the MLPPPO entropy schedule keys, and updates the Phase 6 tracker plus experiments README. ChangesT7 Cross-Architecture Ranking (Experiment 029)
Estimated code review effortðŊ 3 (Moderate) | âąïļ ~25 minutes Possibly related PRs
ðĨ Pre-merge checks | â 4 | â 1â Failed checks (1 warning)
â Passed checks (4 passed)
âïļ Tip: You can configure your own custom pre-merge checks in the settings. âĻ Finishing Touchesð Generate docstrings
𧊠Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ðĪ Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@docs/experiments/logbooks/supporting/029-continuous-architecture-ranking/ga_c3_results.json`:
- Around line 1-34: The GA results JSON is keyed by seed at the top level, but
the ranking ingestion in t7_continuous_ranking.py expects a "results" object, so
this payload is currently ignored. Fix the contract by either wrapping the seed
entries under a top-level "results" key in ga_c3_results.json or updating the
loader logic in the ranking script to accept both the existing top-level schema
and the "results"-wrapped schema. Make the change in the ranking loader path
that uses json.loads(...).get("results", {}) so GA data is preserved.
In `@scripts/analysis/t7_continuous_ranking.py`:
- Around line 61-82: The ranking path in t7_continuous_ranking.py currently
ignores episode budget, so mixed 3000/4000/6000-run manifests can still be
ranked together. Add a budget validation step before computing the plateau tail
in _plateau_tail and the downstream ranking flow (the related logic in the
referenced sections) so only a uniform 6000-episode dataset is accepted, and
fail fast or skip inputs that do not match the expected budget.
- Around line 128-145: The paired-seed ranking logic in the continuous-ranking
script is using partial overlaps, which changes the effective sample size per
comparison and can drop GA silently. Add a validation step in the
results-loading/ranking path (near the code that builds the `table` and before
any pairwise tests) to require a complete seed panel for every arm, including
`feedforwardga`, and fail fast if any seeds are missing. Then compute a single
shared seed intersection once and reuse it for all comparisons instead of
recomputing `common` per pair.
In `@scripts/analysis/t7_ga_champion_eval.py`:
- Around line 63-92: The GA seed scan in the loop over
args.runs_dir.glob("resultsdir_s*.txt") should fail fast when no files are found
instead of writing an empty results object and printing a nan mean. Add an
explicit empty-check before the evaluation loop in the script that builds
results/resultsdir entries, and raise an error or exit with a clear message if
no matching seeds are present so downstream ranking cannot silently proceed with
a missing GA arm.
ðŠ Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
âđïļ Review info
âïļ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 6d5dd27c-1e61-46dc-adce-033110268573
â Files ignored due to path filters (2)
docs/experiments/logbooks/supporting/029-continuous-architecture-ranking/per-behaviour.csvis excluded by!**/*.csvdocs/experiments/logbooks/supporting/029-continuous-architecture-ranking/ranking-per-seed.csvis excluded by!**/*.csv
ð Files selected for processing (9)
configs/scenarios/foraging_predator_thermal/mlpppo_small_continuous2d_combined_klinotaxis.ymldocs/experiments/README.mddocs/experiments/logbooks/029-continuous-architecture-ranking.mddocs/experiments/logbooks/supporting/029-continuous-architecture-ranking/cross_arch_pairwise.jsondocs/experiments/logbooks/supporting/029-continuous-architecture-ranking/details.mddocs/experiments/logbooks/supporting/029-continuous-architecture-ranking/ga_c3_results.jsonopenspec/changes/phase6-tracking/tasks.mdscripts/analysis/t7_continuous_ranking.pyscripts/analysis/t7_ga_champion_eval.py
Comment on lines
+61
to
+82
| def _plateau_tail(out_path: Path) -> tuple[float, float] | None: | ||
| """(full-clear success %, mean foods) over the final-quarter plateau tail, from the .out. | ||
|
|
||
| The robust primary (success) + foraging sub-metric (foods), read from the per-run | ||
| ``.out`` rather than the JSON's ``post_convergence_*`` / ``avg_*`` fields: the | ||
| post-convergence fields fall back to a noisy last-10-run mean for non-converged runs, | ||
| and the ``avg_*`` fields include the warm-up. The plateau-tail mean is the robust | ||
| plateau level (it equals ``post_convergence_success_rate`` within sampling noise for | ||
| converged seeds â the metric's final-window cross-check). | ||
| """ | ||
| if not out_path.exists(): | ||
| return None | ||
| succ, foods = [], [] | ||
| for ln in out_path.read_text().splitlines(): | ||
| m = re.match(r"Run:\s+\d+\s+Status:\s+(\S+).*?Eaten:\s+(\d+)/", ln) | ||
| if m: | ||
| succ.append(1.0 if m.group(1) == "SUCCESS" else 0.0) | ||
| foods.append(int(m.group(2))) | ||
| if not succ: | ||
| return None | ||
| tail = max(1, int(len(succ) * _FINAL_WINDOW_FRAC)) | ||
| return 100.0 * float(np.mean(succ[-tail:])), float(np.mean(foods[-tail:])) |
There was a problem hiding this comment.
ðŊ Functional Correctness | ð Major | ⥠Quick win
Validate the episode budget before ranking.
This script discards run length and even documents mixed budgets as acceptable, so a manifest that blends 3000/4000/6000-episode runs will still produce a ranking. That reintroduces the exact slow-learner under-ranking problem the T7 methodology says was fixed by moving to a uniform 6000-episode budget plus a separate 8000-episode sensitivity pass.
Suggested change
-def _plateau_tail(out_path: Path) -> tuple[float, float] | None:
+def _plateau_tail(out_path: Path) -> tuple[float, float, int] | None:
@@
- return 100.0 * float(np.mean(succ[-tail:])), float(np.mean(foods[-tail:]))
+ return 100.0 * float(np.mean(succ[-tail:])), float(np.mean(foods[-tail:])), len(succ)
@@
- success, foods_tail = tail
+ success, foods_tail, run_count = tail
r = json.loads(jpath.read_text()).get("results", {})
@@
"overall_success": (r.get("success_rate") or 0.0) * 100.0,
"foods": foods_tail,
+ "run_count": run_count,
"evasion_rate": (evas / enc * 100.0) if (enc and enc > 0) else None,
"temp_comfort": r.get("post_convergence_temperature_comfort_score"),
}
@@
for arch in PPO_ARCHS:
missing = [s for s in range(1, 9) if s not in table[arch]]
if missing:
print(f" WARN {arch}: {len(missing)} seed(s) missing experiment JSON {missing}")
+
+ run_counts = {
+ m["run_count"]
+ for arch in PPO_ARCHS
+ for m in table[arch].values()
+ if "run_count" in m
+ }
+ if len(run_counts) != 1:
+ msg = f"Mixed PPO budgets in ranking input: {sorted(run_counts)}"
+ raise ValueError(msg)Also applies to: 85-111, 114-145
ðĪ Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/analysis/t7_continuous_ranking.py` around lines 61 - 82, The ranking
path in t7_continuous_ranking.py currently ignores episode budget, so mixed
3000/4000/6000-run manifests can still be ranked together. Add a budget
validation step before computing the plateau tail in _plateau_tail and the
downstream ranking flow (the related logic in the referenced sections) so only a
uniform 6000-episode dataset is accepted, and fail fast or skip inputs that do
not match the expected budget.
Comment on lines
+128
to
+145
| for arch in PPO_ARCHS: | ||
| missing = [s for s in range(1, 9) if s not in table[arch]] | ||
| if missing: | ||
| print(f" WARN {arch}: {len(missing)} seed(s) missing experiment JSON {missing}") | ||
|
|
||
| if ga_results and ga_results.exists(): | ||
| ga = json.loads(ga_results.read_text()) | ||
| table["feedforwardga"] = { | ||
| int(s): { | ||
| "success": v["full_clear_rate"], | ||
| "converged": None, # n/a â the GA champion eval has no training-plateau notion | ||
| "foods": v.get("mean_foods"), | ||
| "evasion_rate": None, | ||
| "temp_comfort": None, | ||
| } | ||
| for s, v in ga.items() | ||
| } | ||
| return table |
There was a problem hiding this comment.
ðïļ Data Integrity & Integration | ð Major | ⥠Quick win
Reject incomplete seed panels instead of recomputing common per pair.
The published method is a 6-arm, paired-seed n=8 ranking, but this only prints warnings for missing seeds and then runs each pairwise test on whatever overlap happens to exist. That silently changes n between comparisons and can even omit GA entirely while still emitting BH-FDR results.
Suggested guard
for arch in PPO_ARCHS:
missing = [s for s in range(1, 9) if s not in table[arch]]
if missing:
- print(f" WARN {arch}: {len(missing)} seed(s) missing experiment JSON {missing}")
+ msg = f"{arch} is missing paired seeds {missing}"
+ raise ValueError(msg)
@@
if ga_results and ga_results.exists():
ga = json.loads(ga_results.read_text())
table["feedforwardga"] = {
@@
for s, v in ga.items()
}
+ missing = [s for s in range(1, 9) if s not in table["feedforwardga"]]
+ if missing:
+ msg = f"feedforwardga is missing paired seeds {missing}"
+ raise ValueError(msg)
+ else:
+ raise ValueError("ga_results is required for the 6-arm T7 ranking")
@@
for a, b in combinations(sorted(archs, key=lambda x: -means[x][0]), 2):
common = sorted(
set(_per_seed(primary[a], "success")) & set(_per_seed(primary[b], "success")),
)
+ if common != list(range(1, 9)):
+ msg = f"Pair {a} vs {b} is not using the full paired seed panel: {common}"
+ raise ValueError(msg)
deltas = [primary[a][s]["success"] - primary[b][s]["success"] for s in common]Also applies to: 185-202
ðĪ Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/analysis/t7_continuous_ranking.py` around lines 128 - 145, The
paired-seed ranking logic in the continuous-ranking script is using partial
overlaps, which changes the effective sample size per comparison and can drop GA
silently. Add a validation step in the results-loading/ranking path (near the
code that builds the `table` and before any pairwise tests) to require a
complete seed panel for every arm, including `feedforwardga`, and fail fast if
any seeds are missing. Then compute a single shared seed intersection once and
reuse it for all comparisons instead of recomputing `common` per pair.
Two silent-failure gaps from branch review:
- t7_ga_champion_eval.py: raise (instead of writing an empty results JSON +
printing a nan mean) when the runs-dir has no resultsdir_s*.txt pointers.
- t7_continuous_ranking.py: warn (instead of silently excluding the GA arm) when
--ga-results is provided but the file is missing/empty.
Skipped the other review findings as not-valid against current code: the GA
loader reads the top-level seed-keyed dict via ga.items() (NOT the PPO
.get("results") path â GA already ingests correctly, confirmed by the published
ranking); the plateau-tail metric is intentionally budget-agnostic and the
protocol allows non-uniform budgets (the final ranking itself mixes 6000/8000ep);
and per-pair common-seed intersection is the established (025) paired-test
behaviour that uses more data than a single global intersection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Reportâ All modified and coverable lines are covered by tests. ðĒ Thoughts on this report? Let us know! |
This was referenced Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The T7 (high-fidelity continuous-2D) cross-architecture ranking â the analogue of Logbook 025 (T4 grid). Ranks the 6 MUST arms on the integrated C3 cell (food + predator + thermotaxis) at n=8 paired seeds, plateau-tail full-clear, uniform 6000ep + a 5-seed 8000ep convergence top-up, with paired Wilcoxon + 80% bootstrap + BH-FDR (the committed
architecture-comparison-protocolstats layer).Result
MLP 89.0 âŦ {CfC 75.8 â Transformer 74.0} > LSTM 60.1 > connectome 52.2 âŦ GA 15.0 â three significant tiers (BH-FDR), robust across budget levels.
Contents
docs/experiments/logbooks/029-continuous-architecture-ranking.mdâ main logbook (modeled on 025).supporting/029/â analysis outputs only (~28 KB, following 025's lighter practice): per-seed CSV, per-behaviour CSV, pairwise-stats JSON, GA results,details.md(budget-iteration story + the methodology dig-ins).scripts/analysis/t7_continuous_ranking.py+t7_ga_champion_eval.pyâ the ranking harness (reuses the committed Wilcoxon/bootstrap/BH-FDR helpers).c3_integratedcells +analysis.{ranking_upgraded, env_delta, connectome_final_ranking}.entropy_coef_end/entropy_decay_episodeskeys (mlpppo never implemented the schedule; it ran flat 0.08) + relabeled.Methodology (two dig-in fixes the result depended on)
convergedflag.Scope
T7.logbookis left unchecked â T7 is NOT closed. This publishes the MUST ranking; the SHOULD/MAY arms and a memory-bound control are deferred to a follow-up (the box ticks at T7 closure, not here).Validation
openspec validate --strictâ; pre-commit (ruff/pyright/mdformat/markdownlint/tests) â; all cross-doc links resolve; numbers cross-checked (logbook = CSV = pairwise JSON); notmp/links or home paths; no LFS-worthy files.ðĪ Generated with Claude Code
Summary by CodeRabbit