⭐ Must Read: sgl project/sglang/pull/

kflansburg · 2026-07-12T15:06:51Z

Motivation

When grammar validation fails after a disaggregated prefill KV transfer has started, the request remains in the inflight queue until the transfer reaches a terminal state. Releasing its KV cache immediately can release the same allocation again during terminal cleanup. Terminal handling can also replace the original grammar error with a transfer result.

Modifications

Defer KV cache release after grammar rejection until the inflight transfer reaches a terminal state.
Preserve an existing grammar abort reason during successful and failed transfer cleanup.
Add CPU unit coverage for transferring, successful, and failed poll states, plus deferred external abort behavior.

Accuracy Tests

Not applicable. This change only affects cleanup after grammar validation rejects a token.

Speed Tests and Profiling

Not applicable. The normal inference path is unchanged.

Checklist

Format code with pre-commit.
Add unit tests.
Documentation is not required because there are no user-facing API changes.
Accuracy and speed benchmarks are not applicable to this error-path cleanup.
Follow the SGLang code style guidance.

Review and Merge Process

Get approvals from CODEOWNERS and other reviewers.
Trigger the required CI tests.
Merge after CI and required approvals are complete.

CI States

Latest PR Test (Base): ❌ Run #29323853939
Latest PR Test (Extra): ❌ Run #29323853715

gemini-code-assist

Code Review

This pull request modifies the disaggregation prefill queue handling to prevent overwriting an existing FINISH_ABORT status on requests. It also adds a comprehensive unit test suite to verify prefill queue cleanup under grammar failures and deferred aborts. The reviewer feedback highlights a critical issue where external aborts (req.to_finish) are silently discarded because they are not promoted to req.finished_reason on the prefill server. Suggestions are provided to promote req.to_finish to req.finished_reason in both the success and failure paths, and to update the new unit tests accordingly.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-12T15:08:17Z

+                if not isinstance(req.finished_reason, FINISH_ABORT):
+                    req.finished_reason = FINISH_LENGTH(length=0)


If a request is aborted externally, req.to_finish is set to FINISH_ABORT. Since update_finish_state is not called on the prefill server, to_finish is never promoted to finished_reason. Overwriting finished_reason with FINISH_LENGTH here will silently discard the external abort, causing the request to be treated as successful and potentially wasting downstream decode resources. We should promote req.to_finish to req.finished_reason if it is set.

if req.to_finish is not None: req.finished_reason = req.to_finish elif not isinstance(req.finished_reason, FINISH_ABORT): req.finished_reason = FINISH_LENGTH(length=0)

gemini-code-assist · 2026-07-12T15:08:17Z

+                if not isinstance(req.finished_reason, FINISH_ABORT):
+                    prepare_abort(
+                        req, error_message, status_code=HTTPStatus.INTERNAL_SERVER_ERROR
+                    )


Similarly, if the transfer fails but the request was already aborted externally, we should promote req.to_finish to req.finished_reason to preserve the original external abort reason instead of overwriting it with a transfer failure abort.

if req.to_finish is not None: req.finished_reason = req.to_finish elif not isinstance(req.finished_reason, FINISH_ABORT): prepare_abort( req, error_message, status_code=HTTPStatus.INTERNAL_SERVER_ERROR )

gemini-code-assist · 2026-07-12T15:08:17Z

+            )
+
+        self.assertEqual(done, [req])
+        self.assertIsInstance(req.finished_reason, FINISH_LENGTH)


With the fix to promote req.to_finish to req.finished_reason, the request should finish with FINISH_ABORT rather than FINISH_LENGTH when a deferred abort is present.

Suggested change

self.assertIsInstance(req.finished_reason, FINISH_LENGTH)

self.assertIsInstance(req.finished_reason, FINISH_ABORT)

gemini-code-assist · 2026-07-13T21:40:44Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

ShangmingCai

Looks good.

ShangmingCai · 2026-07-14T06:51:05Z

/rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py

ShangmingCai · 2026-07-14T06:51:21Z

/rerun-group disaggregation

github-actions · 2026-07-14T06:51:27Z

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit cbcbef6.

Re-run was not dispatched. What to do:

Rebase your branch onto the latest main and push again
Follow issue #21065 for context
CI-fix PRs may request the bypass-maintenance label to skip this check

github-actions · 2026-07-14T06:51:47Z

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit cbcbef6.

Re-run was not dispatched. What to do:

Rebase your branch onto the latest main and push again
Follow issue #21065 for context
CI-fix PRs may request the bypass-maintenance label to skip this check

ShangmingCai · 2026-07-14T08:23:56Z

/rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py

ShangmingCai · 2026-07-14T08:24:00Z

/rerun-group disaggregation

github-actions · 2026-07-14T08:24:18Z

Results for /rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py:

🚀 ubuntu-latest (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_prefill_queue_cleanup.py

github-actions · 2026-07-14T08:24:26Z

Results for /rerun-group disaggregation:

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_aarch64.py

🚀 2-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py
cd test/ && python3 registered/disaggregation/test_disaggregation_decode_offload.py
cd test/ && python3 registered/disaggregation/test_disaggregation_optimistic_prefill.py

🚀 8-gpu-h20 (4 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_decode_radix_cache.py
cd test/ && python3 registered/disaggregation/test_disaggregation_different_tp.py
cd test/ && python3 registered/disaggregation/test_disaggregation_dp_attention.py
cd test/ && python3 registered/disaggregation/test_disaggregation_pp.py

🚀 8-gpu-h200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dsv4.py
cd test/ && python3 registered/disaggregation/test_disaggregation_hisparse.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_hybrid_attention.py

🚀 1-gpu-5090 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_xpu.py
cd test/ && python3 registered/disaggregation/test_specv2_kvcache_offloading.py

🚀 4-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_epd_disaggregation.py

ShangmingCai · 2026-07-14T08:37:17Z

/rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py

github-actions · 2026-07-14T08:37:42Z

Results for /rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py:

🚀 ubuntu-latest (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_prefill_queue_cleanup.py

ShangmingCai · 2026-07-14T09:51:29Z

/rerun-group disaggregation

ShangmingCai · 2026-07-14T09:51:35Z

/rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py

github-actions · 2026-07-14T09:51:57Z

Results for /rerun-test test/registered/unit/disaggregation/test_prefill_queue_cleanup.py:

🚀 ubuntu-latest (1 test): ❌ View workflow run

cd test/ && python3 registered/unit/disaggregation/test_prefill_queue_cleanup.py

github-actions · 2026-07-14T09:52:01Z

Results for /rerun-group disaggregation:

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_aarch64.py

🚀 2-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_basic.py
cd test/ && python3 registered/disaggregation/test_disaggregation_decode_offload.py
cd test/ && python3 registered/disaggregation/test_disaggregation_optimistic_prefill.py

🚀 8-gpu-h20 (4 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_decode_radix_cache.py
cd test/ && python3 registered/disaggregation/test_disaggregation_different_tp.py
cd test/ && python3 registered/disaggregation/test_disaggregation_dp_attention.py
cd test/ && python3 registered/disaggregation/test_disaggregation_pp.py

🚀 8-gpu-h200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dsv4.py
cd test/ && python3 registered/disaggregation/test_disaggregation_hisparse.py

🚀 8-gpu-h200 (1 test): ❌ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_hybrid_attention.py

🚀 1-gpu-5090 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_xpu.py
cd test/ && python3 registered/disaggregation/test_specv2_kvcache_offloading.py

🚀 4-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_epd_disaggregation.py

…eanup.py

…gl-project#30937) Co-authored-by: Shangming Cai <csmthu@gmail.com>

fix: avoid double KV release on grammar errors

57dc7e6

gemini-code-assist Bot reviewed Jul 12, 2026

View reviewed changes

Jiminator marked this pull request as ready for review July 13, 2026 21:40

Jiminator requested review from ByronHsu, HaiShaw, ShangmingCai, hnyls2002 and sogalin as code owners July 13, 2026 21:40

Jiminator assigned Jiminator, kpham-sgl and ShangmingCai and unassigned Jiminator Jul 14, 2026

ShangmingCai approved these changes Jul 14, 2026

View reviewed changes

ShangmingCai added 2 commits July 14, 2026 15:37

Merge branch 'main' into kflansburg/fix-disagg-grammar-double-release

48dedbf

fix lint after conflict address

463d230

Merge branch 'main' into kflansburg/fix-disagg-grammar-double-release

a303889

fix test

8e6a58d

Deletremove test/registered/unit/disaggregation/test_prefill_queue_cl…

e430926

…eanup.py

ShangmingCai merged commit bbb5702 into sgl-project:main Jul 14, 2026
97 of 113 checks passed

This was referenced Jul 14, 2026

[CI Monitor] Daily Report - 2026-07-14 bingxche/sglang-ci-bot#135

Open

[CI Monitor] Daily Report - 2026-07-15 bingxche/sglang-ci-bot#136

Open

KastanDay mentioned this pull request Jul 22, 2026

[v0.5.15 06/21] Preserve single KV-release ownership after grammar failure KastanDay/sglang#6

Closed

3 of 5 tasks

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

fix: avoid double KV release on disaggregated prefill grammar errors (s…

db528fb

…gl-project#30937) Co-authored-by: Shangming Cai <csmthu@gmail.com>

		if not isinstance(req.finished_reason, FINISH_ABORT):
		req.finished_reason = FINISH_LENGTH(length=0)

	self.assertIsInstance(req.finished_reason, FINISH_LENGTH)
	self.assertIsInstance(req.finished_reason, FINISH_ABORT)

Conversation

kflansburg commented Jul 12, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot commented Jul 13, 2026

Uh oh!

ShangmingCai left a comment

Choose a reason for hiding this comment

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

⚠️ Rebase Required Before Re-run

Uh oh!

github-actions Bot commented Jul 14, 2026

⚠️ Rebase Required Before Re-run

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

ShangmingCai commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kflansburg commented Jul 12, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading