[NematodeBench] Classical REINFORCE MLP on Predator Small - 0.624

[chrisjz]

NematodeBench Submission

Brain Architecture: REINFORCE MLP
Category: predator_small/classical
Composite Score: 0.624 ± 0.123
Success Rate: 73.4% ± 10.9%

Configuration

• Optimization: REINFORCE policy gradient MLP

Approach

REINFORCE MLP using current default config.

Reproducibility

• 10+ independent sessions completed
• 50+ runs per session
• All seeds unique across all runs
• evaluate_submission.py passes
• Config files in artifacts/experiments/
• Leaderboards regenerated

Files Changed

• benchmarks/predator_small/20251229_110151.json
• artifacts/experiments/20251229_110151/ (10+ session folders)
• README.md (Current Leaders section updated)
• docs/nematodebench/LEADERBOARD.md (Full leaderboard updated)

Summary by CodeRabbit

• Documentation

  • Updated benchmark results with new mlp model metrics for Predator Small - Classical environment (Score 0.624 ± 0.123, Success Rate 73.4% ± 10.9%, Learning Speed 0.84 ± 0.09, Stability 0.52 ± 0.19).

• Chores

  • Added comprehensive benchmark data artifacts containing detailed experiment results and performance metrics from multiple experimental runs and configurations.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds one leaderboard entry to README.md documenting an MLP model benchmark result for the Predator Small - Classical environment, alongside ten new JSON benchmark artifact files containing detailed experimental results with metadata, configuration summaries, and 200 run records each.

Changes

Cohort / File(s)	Summary
Documentation Update README.md	Added a new leaderboard row for MLP model with metrics: Score 0.624 ± 0.123, Success Rate 73.4% ± 10.9%, Learning Speed 0.84 ± 0.09, Stability 0.52 ± 0.19, Distance Efficiency 0.39 ± 0.07 (12 sessions, contributed 2025-12-29).
Benchmark Artifacts artifacts/benchmarks/20251229_110151/20251229_100737.json, 20251229_100739.json, 20251229_100742.json, 20251229_100745.json, 20251229_101742.json, 20251229_101745.json, 20251229_101747.json, 20251229_101750.json, 20251229_103141.json	Added 9 JSON benchmark result files containing comprehensive experiment records including metadata (experiment_id, timestamp, config, git info, system specs), configuration summaries (brain_type, environment_type, grid_size, predators_enabled), per-run results arrays with 200 run entries (run, seed, success, steps, reward, termination_reason, foods_collected, distance_efficiency), aggregate statistics (chemotaxis indices, time metrics, path efficiency, convergence data, learning_speed, stability), and biological validation fields (matches_biology, biological_ci_range, literature_source).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• feat: Restructure storing benchmark artifacts #29: Directly related—shares the same benchmark artifact storage structure and README leaderboard update pattern.

Suggested labels

benchmark

Poem

🐰 Ten datasets hop along the way,
With MLP results on display,
Each benchmark run, so neat and tamed,
In JSON rows, forever named!
The leaderboard now shines so bright,
A chemotactic delight! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly describes the main change: adding a classical REINFORCE MLP benchmark result to the Predator Small category with a specific score.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch benchmark/20251229-reinforce-mlp-predator-small

---

📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 359790d and 59859ff.

📒 Files selected for processing (16)

• README.md
• artifacts/benchmarks/20251229_110151/20251229_100737.json
• artifacts/benchmarks/20251229_110151/20251229_100739.json
• artifacts/benchmarks/20251229_110151/20251229_100742.json
• artifacts/benchmarks/20251229_110151/20251229_100745.json
• artifacts/benchmarks/20251229_110151/20251229_101742.json
• artifacts/benchmarks/20251229_110151/20251229_101745.json
• artifacts/benchmarks/20251229_110151/20251229_101747.json
• artifacts/benchmarks/20251229_110151/20251229_101750.json
• artifacts/benchmarks/20251229_110151/20251229_103141.json
• artifacts/benchmarks/20251229_110151/20251229_103144.json
• artifacts/benchmarks/20251229_110151/20251229_103146.json
• artifacts/benchmarks/20251229_110151/20251229_103149.json
• artifacts/benchmarks/20251229_110151/config.yml
• benchmarks/predator_small/classical/20251229_110151.json
• docs/nematodebench/LEADERBOARD.md

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-12-28T15:34:32.857Z

Learnt from: chrisjz
Repo: SyntheticBrains/nematode PR: 28
File: artifacts/experiments/20251228_145113/20251228_145113.json:2052-2067
Timestamp: 2025-12-28T15:34:32.857Z
Learning: In the NematodeBench project, `matches_biology=false` in experiment results is expected at the current stage. No current models match biological behavior yet; the biological validation metrics (chemotaxis_index, matches_biology flag, etc.) are tracked for future purposes once biologically equivalent models are supported.

Applied to files:

• artifacts/benchmarks/20251229_110151/20251229_101747.json
• artifacts/benchmarks/20251229_110151/20251229_100739.json
• artifacts/benchmarks/20251229_110151/20251229_100742.json
• artifacts/benchmarks/20251229_110151/20251229_101750.json
• artifacts/benchmarks/20251229_110151/20251229_101742.json
• artifacts/benchmarks/20251229_110151/20251229_103141.json
• artifacts/benchmarks/20251229_110151/20251229_101745.json
• artifacts/benchmarks/20251229_110151/20251229_100737.json
• artifacts/benchmarks/20251229_110151/20251229_100745.json

🔇 Additional comments (12)

README.md (1)

223-229: New Predator Small – Classical MLP row looks consistent

The added mlp row matches the existing table schema, uses the correct brain label, date, and contributor handle, and is consistent with the reported Predator‑Small/classical artifacts (sessions=12, brain_type=mlp). No changes needed.

artifacts/benchmarks/20251229_110151/20251229_101742.json (1)

1-2069: Artifact schema and aggregates are internally consistent

• experiment_id, path, and exports_path align.
• total_runs matches the 1–200 per_run_results indices, and termination_reason tallies sum exactly to 200.
• High‑level stats (success_rate, composite score, post‑convergence metrics) are numerically plausible for the run breakdown.
• Biological fields (chemotaxis_validation_level="none", matches_biology=false, literature reference) match the expected current project conventions. Based on learnings, this looks correct.

artifacts/benchmarks/20251229_110151/20251229_101747.json (1)

1-2069: Benchmark artifact is well‑formed and consistent with other sessions

The experiment metadata, config summary, and system block match the other mlp_predators_small runs. total_runs, per‑run indices, terminationReason tallies, and success_rate (145/200=0.725) are consistent, and composite/post‑convergence metrics look coherent. Biological validation fields again use chemotaxis_validation_level="none" and matches_biology=false, which is expected at this stage. Based on learnings, this is correct.

artifacts/benchmarks/20251229_110151/20251229_100737.json (1)

1-2069: Session artifact is internally consistent and matches the benchmark schema

Key fields (experiment_id, exports_path, config summary) line up, total_runs matches 200 per‑run entries, and termination_reason counts sum to 200 with success_rate=0.835 (=167/200). Composite and post‑convergence metrics are in the expected range. Biological metrics and matches_biology=false follow the established convention. Based on learnings, this looks good.

artifacts/benchmarks/20251229_110151/20251229_100742.json (1)

1-2069: Third session artifact is consistent with other mlp predator runs

The metadata and config summary match the other mlp_predators_small experiments, and total_runs, index range, and terminationReason counts sum correctly, with success_rate=0.81 (=162/200). Composite, learning_speed, and stability values are in line with the rest of the session set. Biological validation fields (including matches_biology=false) are as expected. Based on learnings, no changes needed.

artifacts/benchmarks/20251229_110151/20251229_101750.json (1)

1-2069: Schema and aggregates look consistent for this benchmark artifact

• Filename, experiment_id, and exports_path are aligned (20251229_101750), and config_* / git metadata match the intended MLP predator-small setup.
• total_runs=200 matches the 200 per_run_results entries; completed_all_food + starved + max_steps_reached + predator_deaths = 200, and success_rate=0.67 matches 134 completions / 200 runs.
• Convergence flags and post‑convergence metrics are populated consistently with converged=true.
• Chemotaxis / biology fields are present and matches_biology=false is in line with current project expectations. Based on learnings, this is the expected state for all current models.

artifacts/benchmarks/20251229_110151/20251229_101745.json (1)

1-2069: Non‑converged session is clearly flagged; please confirm inclusion intent

• Core metadata, total_runs, per‑run entries, and termination counters are internally consistent (99+24+7+70 = 200, success_rate=0.495 from 99 completions).
• converged=false with convergence_run/runs_to_convergence null and stability=0.0 correctly marks this as a non‑converged run, while post_convergence_* metrics are still populated (likely reflecting tail‑segment stats rather than true post‑convergence behavior).
• Biology block and matches_biology=false are consistent with current project practice. Based on learnings, this is expected.

Please just double‑check that your leaderboard / aggregation code is intended to include such non‑converged sessions with their recorded composite_benchmark_score, rather than filtering them out or treating them differently.

artifacts/benchmarks/20251229_110151/20251229_100739.json (1)

1-2069: Well‑formed high‑performing session artifact

• IDs and paths (experiment_id, filename, exports_path) are consistent for 20251229_100739.
• Run accounting is correct (162+7+1+30 = 200, success_rate=0.81 from 162 completions), and total_runs matches the 200 per_run_results.
• Convergence and post‑convergence fields are coherently set for converged=true, and the composite score / stability values are in a plausible range.
• Biology / chemotaxis fields follow the project’s pattern with matches_biology=false, which is expected at this stage. Based on learnings, this is consistent.

artifacts/benchmarks/20251229_110151/20251229_103141.json (1)

1-2069: Benchmark JSON is consistent and matches the established schema

• experiment_id, filename, and exports_path are aligned for 20251229_103141; the config and git metadata match the other predator‑small MLP sessions.
• Run and outcome tallies are consistent (156+8+0+36 = 200, success_rate=0.78 from 156 completions), and total_runs matches the 200 per_run_results.
• converged=true with a reasonable convergence_run and coherent post‑convergence metrics / composite score.
• Chemotaxis / biology fields, including matches_biology=false, are in line with current project expectations. Based on learnings, this is as intended.

artifacts/benchmarks/20251229_110151/20251229_100745.json (3)

2052-2068: Biological validation metrics structure is correct.

The biological validation metrics are properly included with matches_biology: false (line 2066), which is expected at the current stage of the NematodeBench project. The chemotaxis indices, path efficiency metrics, and literature references are tracked for future validation purposes once biologically equivalent models are supported.

Based on learnings, matches_biology=false is expected for current models.

---

18-23: No action needed. The file is correctly configured for the predator_small/classical category.

The benchmark categories distinguish between brain architecture class (quantum vs. classical/MLP) and environment type (predator evasion, foraging, static maze). The term "classical" refers to the MLP brain architecture, not environment configuration. The file correctly shows brain_type: "mlp" (classical) with environment_type: "dynamic" (appropriate for predator evasion tasks) and grid_size: 20 (small category threshold).

---

24-49: This review comment is based on a misunderstanding of the benchmark structure and should be resolved.

The file 20251229_100745.json is ONE individual training session from a 12-session benchmark submission. The metrics in this file (composite score 0.6947, success rate 78%) correctly represent that single session's performance. The PR summary metrics (composite 0.624 ± 0.123, success rate 73.4% ± 10.9%) are the aggregate statistics (mean ± standard deviation) across all 12 independent sessions, as documented in SUBMISSION_GUIDE.md.

Individual sessions naturally vary around the aggregate mean. The file's scores are higher than the mean precisely because this particular session performed better than average—this is expected behavior in multi-session benchmarks, not a discrepancy. All 12 session files in artifacts/benchmarks/20251229_110151/ are correct and contribute to the aggregate leaderboard entry.

Likely an incorrect or invalid review comment.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!