Repository navigation
[Bug] PD disaggregation: aborted prefill requests (input too long) cause decode to generate garbage from incomplete KV cache #30233
Description
Checklist
- I searched related issues but found no solution.
- The bug persists in the latest version.
- Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
- If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- Please use English. Otherwise, it will be closed.
Describe the bug
Bug Description
In PD disaggregation mode, when a request's input length exceeds max_req_input_len (but is under context_len), the prefill side aborts via set_finish_with_abort, which truncates origin_input_ids to 1 token. However, the aborted request still enters the prefill queue, and send_kv_chunk transfers only 1 token of KV cache to the decode side. The decode side, which allocated space for the full prompt length, starts decoding from incomplete/uninitialized KV cache, producing garbage output.
Root Cause
- validate_input_length fails (input > max_req_input_len), allow_auto_truncate=False
- set_finish_with_abort replaces origin_input_ids with array("q", [0]) (1 token placeholder)
- Aborted request still enters prefill queue via _add_request_to_queue
- send_kv_chunk computes end_idx = min(fill_len, len(origin_input_ids)) = min(1, 1) = 1, only sends 1 token of KV cache
- Decode side allocates space for full prompt length (284937) but only receives 1 token of KV data
- Decode starts from 284936 uninitialized KV cache slots â garbage output
Additionally, process_disagg_prefill_inflight_queue overwrites FINISH_ABORT with FINISH_LENGTH(length=0) when KVPoll.Success is received for the 1-token transfer.
Key Parameters
max_total_num_tokens = 284928 (determined by mem_fraction_static)
max_req_len = min(context_len, max_total_num_tokens) = 284928
max_req_input_len = max_req_len - page_size - 1 = 284922
#Request with 284937 tokens > 284922 â triggers abort
Fix
The fix ensures that aborted prefill requests do not transfer KV cache and instead notify the decode side to abort as well:
diff --git a/python/sglang/srt/disaggregation/base/conn.py b/python/sglang/srt/disaggregation/base/conn.py
index a7bf9904a2..a273fc0bbf 100644
--- a/python/sglang/srt/disaggregation/base/conn.py
+++ b/python/sglang/srt/disaggregation/base/conn.py
@@ -95,7 +95,6 @@ class BaseKVManager(ABC):
class BaseKVSender(ABC):
-
@abstractmethod
def __init__(
self,
@@ -151,7 +150,6 @@ class BaseKVSender(ABC):
class BaseKVReceiver(ABC):
-
@abstractmethod
def __init__(
self,
@@ -209,6 +207,13 @@ class BaseKVReceiver(ABC):
"""
pass
+ def notify_decode_abort(self):
+ """
+ Notify the decode side that this request was aborted on prefill side.
+ The decode side should also abort the request.
+ """
+ pass
+
class BaseKVBootstrapServer(ABC):
@abstractmethod
diff --git a/python/sglang/srt/disaggregation/common/conn.py b/python/sglang/srt/disaggregation/common/conn.py
index f025bbd7aa..124398e706 100644
--- a/python/sglang/srt/disaggregation/common/conn.py
+++ b/python/sglang/srt/disaggregation/common/conn.py
@@ -834,6 +834,16 @@ class CommonKVSender(BaseKVSender):
):
pass
+ def notify_decode_abort(self):
+ if self.conclude_state is not None:
+ return
+ self.kv_mgr.record_failure(
+ self.bootstrap_room,
+ "Aborted on prefill side (input validation failed).",
+ )
+ self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)
+ self.conclude_state = KVPoll.Failed
+
def _check_bootstrap_timeout(self) -> Optional[KVPoll]:
if self.init_time is None:
return None
diff --git a/python/sglang/srt/disaggregation/fake/conn.py b/python/sglang/srt/disaggregation/fake/conn.py
index 09c5f80ce4..9933878b6a 100644
--- a/python/sglang/srt/disaggregation/fake/conn.py
+++ b/python/sglang/srt/disaggregation/fake/conn.py
@@ -89,6 +89,9 @@ class FakeKVSender(BaseKVSender):
def abort(self):
self.conclude_state = KVPoll.Failed
+ def notify_decode_abort(self):
+ self.conclude_state = KVPoll.Failed
+
class FakeKVReceiver(BaseKVReceiver):
def __init__(
diff --git a/python/sglang/srt/disaggregation/mooncake/conn.py b/python/sglang/srt/disaggregation/mooncake/conn.py
index b21aee9f7c..4771763ff3 100644
--- a/python/sglang/srt/disaggregation/mooncake/conn.py
+++ b/python/sglang/srt/disaggregation/mooncake/conn.py
@@ -1669,7 +1669,6 @@ class MooncakeKVManager(CommonKVManager):
class MooncakeKVSender(CommonKVSender):
-
def __init__(
self,
mgr: MooncakeKVManager,
@@ -1766,6 +1765,30 @@ class MooncakeKVSender(CommonKVSender):
self.trace_ctx.abort(abort_info={"reason": "Aborted"})
self.trace_ctx.trace_req_finish()
+ def notify_decode_abort(self):
+ if self.conclude_state is not None:
+ return
+ self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)
+ self.conclude_state = KVPoll.Failed
+ prefill_unique_rank = (
+ self.kv_mgr.attn_tp_rank * (self.kv_mgr.pp_size * self.kv_mgr.attn_cp_size)
+ + self.kv_mgr.pp_rank * self.kv_mgr.attn_cp_size
+ + self.kv_mgr.attn_cp_rank
+ )
+ transfer_info_dict = self.kv_mgr.transfer_infos.get(self.bootstrap_room)
+ if transfer_info_dict is not None:
+ for transfer_info in transfer_info_dict.values():
+ if not transfer_info.is_dummy:
+ self.kv_mgr.sync_status_to_decode_endpoint(
+ transfer_info.endpoint,
+ transfer_info.dst_port,
+ self.bootstrap_room,
+ KVPoll.Failed,
+ prefill_unique_rank,
+ )
+ self.trace_ctx.abort(abort_info={"reason": "Aborted on prefill side"})
+ self.trace_ctx.trace_req_finish()
+
class MooncakeKVReceiver(CommonKVReceiver):
def __init__(
diff --git a/python/sglang/srt/disaggregation/mori/conn.py b/python/sglang/srt/disaggregation/mori/conn.py
index 81bb7e960b..539dba3cba 100644
--- a/python/sglang/srt/disaggregation/mori/conn.py
+++ b/python/sglang/srt/disaggregation/mori/conn.py
@@ -1551,9 +1551,12 @@ class MoriKVSender(CommonKVSender):
def abort(self):
self._finalize_failure("Aborted by AbortReq.")
+ def notify_decode_abort(self):
+ if self.conclude_state is not None:
+ return
+ self._finalize_failure("Aborted on prefill side (input validation failed).")
class MoriKVReceiver(CommonKVReceiver):
-
def __init__(
self,
mgr: MoriKVManager,
diff --git a/python/sglang/srt/disaggregation/prefill.py b/python/sglang/srt/disaggregation/prefill.py
index ce1afdac3a..bd51587dfe 100644
--- a/python/sglang/srt/disaggregation/prefill.py
+++ b/python/sglang/srt/disaggregation/prefill.py
@@ -578,6 +578,22 @@ class SchedulerDisaggregationPrefillMixin:
advance_logprob_pt(i, req)
continue
+ # Aborted requests (e.g. input too long): skip KV transfer
+ # and notify decode side to abort as well.
+ if is_aborted(req):
+ logger.warning(
+ f"[ABORT_DEBUG] is_aborted=True, calling notify_decode_abort "
+ f"rid={req.rid} room={req.bootstrap_room}"
+ )
+ req.disagg_kv_sender.notify_decode_abort()
+ release_kv_cache(req, self.tree_cache)
+ req.finished_reason = req.to_finish
+ req.to_finish = None
+ req.time_stats.set_completion_time()
+ self.output_streamer.stream_output([req], req.return_logprob, None)
+ advance_logprob_pt(i, req)
+ continue
+
req.output_ids.append(next_token_id)
maybe_cache_unfinished_req(req, self.tree_cache)
self.disagg_prefill_inflight_queue.append(req)
@@ -688,7 +704,6 @@ class SchedulerDisaggregationPrefillMixin:
undone_reqs: List[Req] = []
# Check .poll() for the reqs in disagg_prefill_inflight_queue. If Success, respond to the client and remove it from the queue
for req, poll in zip(self.disagg_prefill_inflight_queue, polls):
-
if rids_to_check is not None:
if req.rid not in rids_to_check:
undone_reqs.append(req)
@@ -713,7 +728,9 @@ class SchedulerDisaggregationPrefillMixin:
undone_reqs.append(req)
elif poll == KVPoll.Success: # transfer done
release_kv_cache(req, self.tree_cache) # unlock the tree
- req.finished_reason = FINISH_LENGTH(length=0)
+ # Do not overwrite abort status with FINISH_LENGTH
+ if not isinstance(req.finished_reason, FINISH_ABORT):
+ req.finished_reason = FINISH_LENGTH(length=0)
# FIXME: clean up req's data in transfer engine
if hasattr(req.disagg_kv_sender, "clear"):
req.disagg_kv_sender.clear()Reproduction
- Start a PD disaggregation deployment with mooncake transfer backend
- Set mem_fraction_static such that max_total_num_tokens is relatively small (e.g., 284928)
- Send a request with input length slightly exceeding max_req_input_len (e.g., 284937 tokens)
- Observe: prefill returns prompt_tokens=1, decode returns garbage content unrelated to the input
Environment
- SGLang version: 0.5.11
- Model: zai-org/GLM-5.2-FP8 (DeepSeek-V3 architecture, MLA attention, MoE)
- Transfer backend: mooncake
- dp_size=32, tp_size=32, ep_size=32
- mem_fraction_static=0.75, page_size=64
- kv_cache_dtype=fp8_e4m3
- enable_hierarchical_cache=true, hicache_ratio=2, hicache_size=200
- disaggregation_mode=prefill
- disaggregation_transfer_backend=mooncake
- load_balance_method=follow_bootstrap_room
- chunked_prefill_size=131072
- context_len=1048576
- max_total_num_tokens=284928
Activity
github-actions commentedon Sep 5, 2026
This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.