Skip to content

[Fix] Run KV canary hooks for context-parallel prefill - #40642

Merged
ch-wan merged 2 commits into
mainfrom
cheng/hot-fix/canary-cp-forward
Sep 21, 2026
Merged

ch-wan merged 2 commits into
mainfrom
cheng/hot-fix/canary-cp-forward

Conversation

@ch-wan

@ch-wan ch-wan commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

CP prefill calls the transformer body directly and bypasses the outer model.forward patched by KV canary. This skips the canary pre/post hooks and triggers SimplePhaseChecker: phase mismatch (expect=3, actual=1) in both log and raise modes.

When prefill CP is enabled, also wrap the transformer body's forward with the existing canary hooks. The shared bracket scope keeps decode, which still enters through the outer model, from running those hooks twice. Add a CP=2 regression covering prefill followed by CUDA-graph decode in raise mode with real-KV hashing enabled. The registered test starts the server once; log uses the same hook path and was checked separately.

Validation on two H200 GPUs:

  • Reproduced the phase assertion on unmodified main at 66f19f5c46; the same four-request smoke passes with this fix.
  • test/registered/mock_model/test_e2e_cp.py: passed in 46.5 seconds (down from 93.1 seconds for two launches), with token-oracle input checking and --kv-canary-real-data all. A separate log smoke also passed; CI retains only raise to avoid repeating server startup.
  • Qwen3-0.6B real weights and regular sampling: eight requests passed with CP=2, log, real-KV hashing, and verify-token assertions, including repeated prompts that reuse the prefix cache.
  • All applicable pre-commit checks passed.

CI States

Latest PR Test (Base): ❌ Run #35664946928
Latest PR Test (Extra): ❌ Run #35664946646
Latest PR Test (AMD ROCm 10): ❌ Run #35664947008

@ch-wan

ch-wan commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_e2e_cp.py

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_e2e_cp.py:

🚀 2-gpu-h100 (1 test): ❌ View workflow run

cd test/ && python3 registered/mock_model/test_e2e_cp.py

@ch-wan
ch-wan merged commit 22587fb into main Sep 21, 2026
101 of 118 checks passed
@ch-wan
ch-wan deleted the cheng/hot-fix/canary-cp-forward branch September 21, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant