perf: Optimise thermotaxis simulations - #44
Conversation
Decently balances foraging vs navigating dangerous temperature zones
Baslines for all medium configs
📝 WalkthroughWalkthroughAdds nine PPO thermotaxis baseline configs (3 tasks × 3 sizes), implements safe-zone biased food spawning and reward fields, extends sensory classical features (thermotaxis includes temperature deviation), applies brave-foraging and temperature HP-damage into runner rewards, and updates tests, docs, and example configs. Changes
Sequence Diagram(s)mermaid ConfigLoader->>Env: load ForagingParams (safe_zone_food_bias) and ThermotaxisParams (spots, reward_discomfort_food) Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Fix all issues with AI agents
In
`@artifacts/logbooks/007/ppo_foraging_medium/ppo_thermotaxis_foraging_medium.yml`:
- Around line 75-77: Mismatch: target_foods_to_collect (20) is greater than
foods_on_grid (15), forcing respawn waits that contradict the comment about
"More food available...". Fix by either increasing the foraging.foods_on_grid
value to at least match foraging.target_foods_to_collect (or higher) so agents
don't rely on respawns, or change foraging.target_foods_to_collect down to <=
foods_on_grid; alternatively update the descriptive comment to explicitly state
that respawn/waiting is expected for medium difficulty. Reference the keys
target_foods_to_collect and foods_on_grid and ensure the comment describing food
availability is updated to reflect the chosen behavior.
In
`@artifacts/logbooks/007/ppo_pursuit_large/ppo_thermotaxis_pursuit_predators_large.yml`:
- Around line 18-20: The logbook header claims "Food biased toward safe
temperature zones" but the foraging configuration does not set the
safe_zone_food_bias key (it currently defaults to 0.0); either add the
safe_zone_food_bias field under the foraging section with the intended numeric
bias (e.g., >0) or edit the header/comment to remove the claim—apply the same
fix for the other occurrence referenced around lines 91-98 so the header and the
foraging.safe_zone_food_bias setting are consistent.
In `@configs/examples/ppo_thermotaxis_pursuit_predators_large.yml`:
- Around line 91-95: Header claims "Food biased toward safe temperature zones"
but the foraging block lacks the safe_zone_food_bias setting (defaults to 0.0).
Add an explicit safe_zone_food_bias key in the foraging section (e.g.,
safe_zone_food_bias: 0.8 or another appropriate non-zero value) to align the
config with the header, or alternatively update the header text to remove the
bias claim if you prefer uniform distribution.
In `@configs/examples/ppo_thermotaxis_stationary_predators_medium.yml`:
- Around line 86-87: The config sets foods_on_grid: 15 but
target_foods_to_collect: 20, which is inconsistent; either make
target_foods_to_collect <= foods_on_grid or increase foods_on_grid to match if
respawning is unintended, or add an explicit comment/flag indicating respawn
behavior is expected for medium difficulty; update the keys foods_on_grid and
target_foods_to_collect in ppo_thermotaxis_stationary_predators_medium.yml
accordingly so the intended behavior is clear and consistent.
In `@configs/examples/ppo_thermotaxis_stationary_predators_small.yml`:
- Around line 110-115: The config sets comfort_reward and discomfort_penalty to
0.0 which removes reward pressure for the stated “>60% comfort time” success
criterion; either restore a small shaping term (e.g., set comfort_reward > 0
and/or discomfort_penalty < 0) in the defaults or update the success-criteria
text to no longer imply reward-driven behavior, and add a clear documented
default and description for reward_discomfort_food alongside other config
defaults; update the config defaults and the documentation block so explicit
overrides remain visible and consistent (look for the comfort_reward,
discomfort_penalty, and reward_discomfort_food entries to change and document).
In `@docs/experiments/logbooks/007-ppo-thermotaxis-baselines.md`:
- Around line 503-506: The document contains an internal inconsistency: the
"Small/medium need comfort_reward: 0.05" entry (the numbered list around the
"Gradient decay matters" block) contradicts the "Final Configuration Summary"
that shows comfort_reward: 0.0; update the numbered list item to read
"Small/medium need comfort_reward: 0.0" (or add a short parenthetical note that
0.05 was an earlier iteration) so the "Small/medium need comfort_reward"
statement matches the actual config files (e.g.,
ppo_thermotaxis_foraging_small.yml) and the Final Configuration Summary.
In `@scripts/run_simulation.py`:
- Around line 243-251: Move the logging configuration block so it runs before
any log emissions (specifically before the first logger.info call that prints
the simulation seed); configure logger.disabled when log_level == "NONE" or set
logger.setLevel(log_level) and update each FileHandler's level (check
isinstance(handler, logging.FileHandler)) otherwise. Update the code around the
logger variable (the logging setup block currently shown) to execute before the
seed/info log to ensure --log-level NONE suppresses that output.
♻️ Duplicate comments (4)
configs/examples/ppo_thermotaxis_pursuit_predators_small.yml (2)
66-66: Same check on disabling damage penalty.
Line 66 mirrors the 0.0penalty_health_damagechange; please confirm it’s intended here as well.
111-116: Thermotaxis shaping neutralization noted.
Same concern as the stationary-small config about neutral comfort/discomfort shaping vs the success-criteria text.configs/examples/ppo_thermotaxis_foraging_small.yml (2)
55-55: Same check on disabling damage penalty.
Line 55 setspenalty_health_damageto 0.0; confirming intent as in the other small baselines.
90-97: Thermotaxis shaping neutralization noted.
Same note as earlier small configs regarding comfort/discomfort shaping being zeroed.
🧹 Nitpick comments (5)
packages/quantum-nematode/tests/quantumnematode_tests/agent/test_runners.py (5)
660-663: Weak assertion doesn't validate brave bonus behavior.The assertion
assert agent._episode_tracker.rewards > 0 or agent._metrics_tracker.foods_collected > 0is too permissive. It passes if either condition is true, but doesn't verify that the brave bonus was actually applied when food was collected in a discomfort zone.Consider strengthening to verify the bonus was applied:
- if agent._metrics_tracker.foods_collected > 0: - # Food collected means reward_goal was applied per food - # With reward_goal=2.0 and brave_bonus=0.5, each food gives at least 2.5 - assert agent._episode_tracker.rewards > 0 or agent._metrics_tracker.foods_collected > 0 + # If food was collected in discomfort zone, verify rewards include brave bonus + foods = agent._metrics_tracker.foods_collected + if foods > 0: + # Each food gives reward_goal (2.0) + brave_bonus (0.5) = 2.5 minimum + # Minus step penalties, rewards should still be positive with food collected + assert agent._episode_tracker.rewards > -foods * 2.5, ( + f"Expected positive trend with {foods} foods collected at 2.5 each" + )
703-704:assert Trueis a no-op that doesn't validate zone handling.This test claims to verify "comfort zone logic correctly skips brave bonus" but the assertion
assert Truevalidates nothing. If the brave bonus were incorrectly applied in comfort zones, this test would still pass.Consider adding a meaningful assertion:
- # Test passes if no errors - comfort zone logic correctly skips brave bonus - assert True + # Verify brave bonus was NOT applied (only reward_goal per food, no bonus) + # In comfort zone, each food gives exactly reward_goal (2.0) + if agent._metrics_tracker.foods_collected > 0: + # Check that total rewards align with food rewards minus step penalties + # Without brave bonus, rewards should be lower than with it + pass # Or implement specific reward tracking for brave bonusAt minimum, consider tracking whether the brave bonus code path was executed.
738-740: Same issue:assert Truedoesn't validate danger zone behavior.This test should verify that the brave bonus is not applied in danger zones, but
assert Truedoesn't check anything.
774-777: Same issue:assert Truedoesn't validate disabled bonus behavior.When
reward_discomfort_food=0.0, the test should verify no bonus is applied, not just that no exception occurs.
571-618: Test logic validates penalty presence but not magnitude relationship.The test comment says "more HP damage results in more penalty applications" but lines 615-618 only check that both rewards are negative. The test doesn't actually verify that
reward_high_damage < reward_low_damage(i.e., more damage = more negative reward).If the penalty is per-step regardless of damage amount, the assertion is correct but the test name/comment is misleading. Consider:
- # Both should be negative (taking damage), and both should apply penalty per step - # The penalty is per-step when damage > 0, so with same steps, penalties are similar - assert reward_low_damage < 0, "Should have negative reward from HP damage penalty" - assert reward_high_damage < 0, "Should have negative reward from HP damage penalty" + # Both should be negative (taking damage) + # penalty_health_damage is applied per-step when damage > 0, regardless of amount + assert reward_low_damage < 0, "Should have negative reward from HP damage penalty" + assert reward_high_damage < 0, "Should have negative reward from HP damage penalty" + # Note: Penalties are similar because they're per-step, not proportional to damageOr rename the test to
test_temperature_hp_damage_penalty_applied_regardless_of_amount.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@configs/examples/ppo_thermotaxis_foraging_medium.yml`:
- Around line 59-60: The config sets reward_health_gain: 0.1 while
penalty_health_damage: 0.0 which effectively rewards taking damage then healing;
update the baseline by setting penalty_health_damage to a non-zero value (e.g.,
a small negative like 0.1 or a value that balances reward_health_gain) to
enforce the intended damage penalty, or alternatively add a clear
comment/documentation near reward_health_gain and penalty_health_damage stating
that damage is intentionally unpenalized in this baseline and why.
In `@configs/examples/ppo_thermotaxis_stationary_predators_large.yml`:
- Around line 119-127: The hot_spots and cold_spots temperature values are
inconsistent with cultivation_temperature (20.0); update the entries under
hot_spots and cold_spots so all hot_spots have temperatures > 20.0 (e.g.,
22.0–35.0) and all cold_spots have temperatures < 20.0 (e.g., 5.0–15.0);
specifically, change the cold_spots entry [25, 25, 25.0] to a value below 20.0
and change any hot_spots equal to 20.0 to >20.0 so the arrays hot_spots,
cold_spots, and the cultivation_temperature setting are consistent.
🧹 Nitpick comments (5)
docs/experiments/logbooks/007-ppo-thermotaxis-baselines.md (1)
108-115: Parameter table may cause confusion with tuned values.The table shows
danger_hp_damageas 2.0 for Small and 1.5 for Medium, but the detailed results sections document that these were tuned down (e.g., "Reduceddanger_hp_damage: 1.5 → 1.0" for Medium configs at lines 207, 223, 239).Consider either:
- Updating the table to show final tuned values, or
- Adding a note that these are initial values before tuning
This would help readers understand which values to use when replicating the experiments.
packages/quantum-nematode/tests/quantumnematode_tests/agent/test_runners.py (4)
564-570: Consider isolating the HP damage penalty assertion.The test combines
danger_penalty=-0.3withpenalty_health_damage=0.5, making it impossible to verify the HP damage penalty specifically contributed to the negative reward. For stronger test isolation, consider settingdanger_penalty=0.0(similar to the next test) to isolate thepenalty_health_damageeffect.♻️ Suggested improvement
thermotaxis=ThermotaxisParams( enabled=True, cultivation_temperature=20.0, base_temperature=33.0, # Danger zone (30-35°C) gradient_strength=0.0, danger_hp_damage=5.0, # 5 HP damage per step - danger_penalty=-0.3, # Zone penalty + danger_penalty=0.0, # No zone penalty to isolate HP damage penalty ),
616-619: Test could verify the penalty-per-step claim.The comment states the penalty is "applied per-step when damage > 0, regardless of amount," but the test only verifies both scenarios produce negative rewards. To confirm the penalty is indeed independent of damage amount (not proportional), consider asserting that both rewards are approximately equal, or if it's meant to be proportional, that higher damage produces more negative rewards.
♻️ Suggested assertion to verify fixed vs proportional penalty
# Both should be negative (taking damage) # The penalty_health_damage is applied per-step when damage > 0, regardless of amount assert reward_low_damage < 0, "Should have negative reward from HP damage penalty" assert reward_high_damage < 0, "Should have negative reward from HP damage penalty" + # If penalty is fixed per-step (not proportional), rewards should be similar + # If proportional, high damage should be more negative + # Uncomment the appropriate assertion based on intended behavior: + # assert abs(reward_low_damage - reward_high_damage) < 0.1, "Fixed penalty per step" + # assert reward_high_damage < reward_low_damage, "Proportional penalty"
622-668: Tests verify execution but not reward values.The brave foraging tests effectively validate that the code paths execute without errors and that zone detection works correctly. However, they don't verify that the
reward_discomfort_foodbonus is actually applied to rewards when food is collected in discomfort zones (or excluded in comfort/danger zones).For stronger behavioral verification, consider capturing reward values before and after food collection, or comparing total rewards between episodes with and without the bonus configured.
751-788: Test verifies config immutability rather than behavioral correctness.This test confirms that
reward_discomfort_foodremains0.0after the episode, which is useful for verifying config isn't accidentally mutated. However, it doesn't verify that no bonus was actually applied to rewards during food collection.To strengthen this test, consider comparing total rewards between two episodes—one with the bonus disabled and one enabled—when food is collected in the same discomfort zone conditions.
| hot_spots: | ||
| - [75, 50, 25.0] # Hot zone right-center | ||
| - [25, 75, 20.0] # Hot zone upper-left | ||
| - [80, 80, 22.0] # Hot zone upper-right | ||
| # Scattered cold spots | ||
| cold_spots: | ||
| - [25, 25, 25.0] # Cold zone lower-left | ||
| - [75, 25, 20.0] # Cold zone lower-right | ||
| - [50, 85, 18.0] # Cold zone top-center |
There was a problem hiding this comment.
Temperature values for hot/cold spots appear incorrectly configured.
The spot temperatures don't match their designations:
- cold_spots have temps: 25.0°C, 20.0°C, 18.0°C — only 18.0°C is actually below base temperature (20°C)
- hot_spots have temps: 25.0°C, 20.0°C, 22.0°C — one spot is exactly at base temperature (20°C)
The cold spot at [25, 25, 25.0] is actually 5°C warmer than base, making it a hot spot. This contradicts both the inline comments and the header documentation (lines 12-14).
Expected pattern for spots relative to cultivation_temperature: 20.0:
- Hot spots should have temperatures > 20°C (e.g., 25-35°C range)
- Cold spots should have temperatures < 20°C (e.g., 5-15°C range)
🔧 Suggested fix
# Scattered hot spots
hot_spots:
- - [75, 50, 25.0] # Hot zone right-center
- - [25, 75, 20.0] # Hot zone upper-left
- - [80, 80, 22.0] # Hot zone upper-right
+ - [75, 50, 30.0] # Hot zone right-center
+ - [25, 75, 28.0] # Hot zone upper-left
+ - [80, 80, 32.0] # Hot zone upper-right
# Scattered cold spots
cold_spots:
- - [25, 25, 25.0] # Cold zone lower-left
- - [75, 25, 20.0] # Cold zone lower-right
- - [50, 85, 18.0] # Cold zone top-center
+ - [25, 25, 10.0] # Cold zone lower-left
+ - [75, 25, 12.0] # Cold zone lower-right
+ - [50, 85, 8.0] # Cold zone top-center📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| hot_spots: | |
| - [75, 50, 25.0] # Hot zone right-center | |
| - [25, 75, 20.0] # Hot zone upper-left | |
| - [80, 80, 22.0] # Hot zone upper-right | |
| # Scattered cold spots | |
| cold_spots: | |
| - [25, 25, 25.0] # Cold zone lower-left | |
| - [75, 25, 20.0] # Cold zone lower-right | |
| - [50, 85, 18.0] # Cold zone top-center | |
| hot_spots: | |
| - [75, 50, 30.0] # Hot zone right-center | |
| - [25, 75, 28.0] # Hot zone upper-left | |
| - [80, 80, 32.0] # Hot zone upper-right | |
| # Scattered cold spots | |
| cold_spots: | |
| - [25, 25, 10.0] # Cold zone lower-left | |
| - [75, 25, 12.0] # Cold zone lower-right | |
| - [50, 85, 8.0] # Cold zone top-center |
🤖 Prompt for AI Agents
In `@configs/examples/ppo_thermotaxis_stationary_predators_large.yml` around lines
119 - 127, The hot_spots and cold_spots temperature values are inconsistent with
cultivation_temperature (20.0); update the entries under hot_spots and
cold_spots so all hot_spots have temperatures > 20.0 (e.g., 22.0–35.0) and all
cold_spots have temperatures < 20.0 (e.g., 5.0–15.0); specifically, change the
cold_spots entry [25, 25, 25.0] to a value below 20.0 and change any hot_spots
equal to 20.0 to >20.0 so the arrays hot_spots, cold_spots, and the
cultivation_temperature setting are consistent.
There was a problem hiding this comment.
The third value is an intensity (temperature delta), not an absolute temperature. Looking at the code in temperature.py:
Hot spots: temp += intensity * exp(-distance/decay) — adds up to the intensity value
Cold spots: temp -= intensity * exp(-distance/decay) — subtracts up to the intensity value
So with base_temperature: 20.0:
Cold spot [25, 25, 25.0] creates temps down to -5°C at center (20 - 25)
Cold spot [75, 25, 20.0] creates temps down to 0°C at center (20 - 20)
Cold spot [50, 85, 18.0] creates temps down to 2°C at center (20 - 18)
The comments were indeed confusing though — I've updated them to clarify that these are intensity deltas, not absolute temperatures.
There was a problem hiding this comment.
@chrisjz, thank you for the clarification! You're absolutely right—I misunderstood the semantics. The third value is indeed an intensity delta applied exponentially from the spot center, not an absolute temperature.
Your calculations confirm the configuration is correct:
- Cold spots subtract intensity:
[25, 25, 25.0]→ 20 - 25 = -5°C at center ✓ - Hot spots add intensity:
[75, 50, 25.0]→ 20 + 25 = 45°C at center ✓
The updated comments in the file make this much clearer now with the explicit note about deltas and the exponential decay. My apologies for the confusion!
✏️ Learnings added
Learnt from: chrisjz
Repo: SyntheticBrains/nematode PR: 44
File: configs/examples/ppo_thermotaxis_stationary_predators_large.yml:0-0
Timestamp: 2026-01-26T07:21:11.709Z
Learning: In thermotaxis configuration files (e.g., configs/examples/ppo_thermotaxis_*.yml), hot_spots and cold_spots use the format [x, y, intensity] where intensity is a temperature DELTA from base_temperature, not an absolute temperature. Hot spots ADD intensity: temp += intensity * exp(-distance/decay). Cold spots SUBTRACT intensity: temp -= intensity * exp(-distance/decay). For example, with base_temperature 20.0, a cold spot [25, 25, 25.0] creates temperatures down to -5°C at its center (20 - 25), and a hot spot [75, 50, 25.0] creates temperatures up to 45°C at its center (20 + 25).
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
🧠 Learnings used
Learnt from: chrisjz
Repo: SyntheticBrains/nematode PR: 44
File: configs/examples/ppo_thermotaxis_foraging_medium.yml:59-60
Timestamp: 2026-01-26T07:09:11.198Z
Learning: In thermotaxis baseline configs (foraging, pursuit, stationary scenarios), penalty_health_damage was empirically found to perform better at 0.0 based on logbook 007 benchmarking results, despite the intuition that penalizing damage might improve learning.
Learnt from: chrisjz
Repo: SyntheticBrains/nematode PR: 44
File: artifacts/logbooks/007/ppo_foraging_medium/ppo_thermotaxis_foraging_medium.yml:75-77
Timestamp: 2026-01-26T05:54:41.207Z
Learning: In thermotaxis configuration files, it's acceptable for `target_foods_to_collect` to exceed `foods_on_grid` because food respawns dynamically during the episode, ensuring new food becomes available as the agent collects toward the target.
perf: Optimise thermotaxis simulations
1 participant
Changes:
Addresses #37
Summary by CodeRabbit
New Features
Behavior / Perception
Tests
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.