Skip to content

[Bug] PD disaggregation: aborted prefill requests (input too long) cause decode to generate garbage from incomplete KV cache #30233

Closed

Description

@kevin-14

Checklist

  • I searched related issues but found no solution.
  • The bug persists in the latest version.
  • Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
  • If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
  • Please use English. Otherwise, it will be closed.

Describe the bug

Bug Description

In PD disaggregation mode, when a request's input length exceeds max_req_input_len (but is under context_len), the prefill side aborts via set_finish_with_abort, which truncates origin_input_ids to 1 token. However, the aborted request still enters the prefill queue, and send_kv_chunk transfers only 1 token of KV cache to the decode side. The decode side, which allocated space for the full prompt length, starts decoding from incomplete/uninitialized KV cache, producing garbage output.

Root Cause

  1. validate_input_length fails (input > max_req_input_len), allow_auto_truncate=False
  2. set_finish_with_abort replaces origin_input_ids with array("q", [0]) (1 token placeholder)
  3. Aborted request still enters prefill queue via _add_request_to_queue
  4. send_kv_chunk computes end_idx = min(fill_len, len(origin_input_ids)) = min(1, 1) = 1, only sends 1 token of KV cache
  5. Decode side allocates space for full prompt length (284937) but only receives 1 token of KV data
  6. Decode starts from 284936 uninitialized KV cache slots → garbage output
    Additionally, process_disagg_prefill_inflight_queue overwrites FINISH_ABORT with FINISH_LENGTH(length=0) when KVPoll.Success is received for the 1-token transfer.

Key Parameters

max_total_num_tokens = 284928 (determined by mem_fraction_static)
max_req_len = min(context_len, max_total_num_tokens) = 284928
max_req_input_len = max_req_len - page_size - 1 = 284922
#Request with 284937 tokens > 284922 → triggers abort

Fix

The fix ensures that aborted prefill requests do not transfer KV cache and instead notify the decode side to abort as well:

diff --git a/python/sglang/srt/disaggregation/base/conn.py b/python/sglang/srt/disaggregation/base/conn.py
index a7bf9904a2..a273fc0bbf 100644
--- a/python/sglang/srt/disaggregation/base/conn.py
+++ b/python/sglang/srt/disaggregation/base/conn.py
@@ -95,7 +95,6 @@ class BaseKVManager(ABC):


 class BaseKVSender(ABC):
-
     @abstractmethod
     def __init__(
         self,
@@ -151,7 +150,6 @@ class BaseKVSender(ABC):


 class BaseKVReceiver(ABC):
-
     @abstractmethod
     def __init__(
         self,
@@ -209,6 +207,13 @@ class BaseKVReceiver(ABC):
         """
         pass

+    def notify_decode_abort(self):
+        """
+        Notify the decode side that this request was aborted on prefill side.
+        The decode side should also abort the request.
+        """
+        pass
+

 class BaseKVBootstrapServer(ABC):
     @abstractmethod
diff --git a/python/sglang/srt/disaggregation/common/conn.py b/python/sglang/srt/disaggregation/common/conn.py
index f025bbd7aa..124398e706 100644
--- a/python/sglang/srt/disaggregation/common/conn.py
+++ b/python/sglang/srt/disaggregation/common/conn.py
@@ -834,6 +834,16 @@ class CommonKVSender(BaseKVSender):
     ):
         pass

+    def notify_decode_abort(self):
+        if self.conclude_state is not None:
+            return
+        self.kv_mgr.record_failure(
+            self.bootstrap_room,
+            "Aborted on prefill side (input validation failed).",
+        )
+        self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)
+        self.conclude_state = KVPoll.Failed
+
     def _check_bootstrap_timeout(self) -> Optional[KVPoll]:
         if self.init_time is None:
             return None
diff --git a/python/sglang/srt/disaggregation/fake/conn.py b/python/sglang/srt/disaggregation/fake/conn.py
index 09c5f80ce4..9933878b6a 100644
--- a/python/sglang/srt/disaggregation/fake/conn.py
+++ b/python/sglang/srt/disaggregation/fake/conn.py
@@ -89,6 +89,9 @@ class FakeKVSender(BaseKVSender):
     def abort(self):
         self.conclude_state = KVPoll.Failed

+    def notify_decode_abort(self):
+        self.conclude_state = KVPoll.Failed
+

 class FakeKVReceiver(BaseKVReceiver):
     def __init__(
diff --git a/python/sglang/srt/disaggregation/mooncake/conn.py b/python/sglang/srt/disaggregation/mooncake/conn.py
index b21aee9f7c..4771763ff3 100644
--- a/python/sglang/srt/disaggregation/mooncake/conn.py
+++ b/python/sglang/srt/disaggregation/mooncake/conn.py
@@ -1669,7 +1669,6 @@ class MooncakeKVManager(CommonKVManager):


 class MooncakeKVSender(CommonKVSender):
-
     def __init__(
         self,
         mgr: MooncakeKVManager,
@@ -1766,6 +1765,30 @@ class MooncakeKVSender(CommonKVSender):
         self.trace_ctx.abort(abort_info={"reason": "Aborted"})
         self.trace_ctx.trace_req_finish()

+    def notify_decode_abort(self):
+        if self.conclude_state is not None:
+            return
+        self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed)
+        self.conclude_state = KVPoll.Failed
+        prefill_unique_rank = (
+            self.kv_mgr.attn_tp_rank * (self.kv_mgr.pp_size * self.kv_mgr.attn_cp_size)
+            + self.kv_mgr.pp_rank * self.kv_mgr.attn_cp_size
+            + self.kv_mgr.attn_cp_rank
+        )
+        transfer_info_dict = self.kv_mgr.transfer_infos.get(self.bootstrap_room)
+        if transfer_info_dict is not None:
+            for transfer_info in transfer_info_dict.values():
+                if not transfer_info.is_dummy:
+                    self.kv_mgr.sync_status_to_decode_endpoint(
+                        transfer_info.endpoint,
+                        transfer_info.dst_port,
+                        self.bootstrap_room,
+                        KVPoll.Failed,
+                        prefill_unique_rank,
+                    )
+        self.trace_ctx.abort(abort_info={"reason": "Aborted on prefill side"})
+        self.trace_ctx.trace_req_finish()
+

 class MooncakeKVReceiver(CommonKVReceiver):
     def __init__(
diff --git a/python/sglang/srt/disaggregation/mori/conn.py b/python/sglang/srt/disaggregation/mori/conn.py
index 81bb7e960b..539dba3cba 100644
--- a/python/sglang/srt/disaggregation/mori/conn.py
+++ b/python/sglang/srt/disaggregation/mori/conn.py
@@ -1551,9 +1551,12 @@ class MoriKVSender(CommonKVSender):
     def abort(self):
         self._finalize_failure("Aborted by AbortReq.")

+    def notify_decode_abort(self):
+        if self.conclude_state is not None:
+            return
+        self._finalize_failure("Aborted on prefill side (input validation failed).")

 class MoriKVReceiver(CommonKVReceiver):
-
     def __init__(
         self,
         mgr: MoriKVManager,
diff --git a/python/sglang/srt/disaggregation/prefill.py b/python/sglang/srt/disaggregation/prefill.py
index ce1afdac3a..bd51587dfe 100644
--- a/python/sglang/srt/disaggregation/prefill.py
+++ b/python/sglang/srt/disaggregation/prefill.py
@@ -578,6 +578,22 @@ class SchedulerDisaggregationPrefillMixin:
                         advance_logprob_pt(i, req)
                         continue

+                # Aborted requests (e.g. input too long): skip KV transfer
+                # and notify decode side to abort as well.
+                if is_aborted(req):
+                    logger.warning(
+                        f"[ABORT_DEBUG] is_aborted=True, calling notify_decode_abort "
+                        f"rid={req.rid} room={req.bootstrap_room}"
+                    )
+                    req.disagg_kv_sender.notify_decode_abort()
+                    release_kv_cache(req, self.tree_cache)
+                    req.finished_reason = req.to_finish
+                    req.to_finish = None
+                    req.time_stats.set_completion_time()
+                    self.output_streamer.stream_output([req], req.return_logprob, None)
+                    advance_logprob_pt(i, req)
+                    continue
+
                 req.output_ids.append(next_token_id)
                 maybe_cache_unfinished_req(req, self.tree_cache)
                 self.disagg_prefill_inflight_queue.append(req)
@@ -688,7 +704,6 @@ class SchedulerDisaggregationPrefillMixin:
         undone_reqs: List[Req] = []
         # Check .poll() for the reqs in disagg_prefill_inflight_queue. If Success, respond to the client and remove it from the queue
         for req, poll in zip(self.disagg_prefill_inflight_queue, polls):
-
             if rids_to_check is not None:
                 if req.rid not in rids_to_check:
                     undone_reqs.append(req)
@@ -713,7 +728,9 @@ class SchedulerDisaggregationPrefillMixin:
                 undone_reqs.append(req)
             elif poll == KVPoll.Success:  # transfer done
                 release_kv_cache(req, self.tree_cache)  # unlock the tree
-                req.finished_reason = FINISH_LENGTH(length=0)
+                # Do not overwrite abort status with FINISH_LENGTH
+                if not isinstance(req.finished_reason, FINISH_ABORT):
+                    req.finished_reason = FINISH_LENGTH(length=0)
                 # FIXME: clean up req's data in transfer engine
                 if hasattr(req.disagg_kv_sender, "clear"):
                     req.disagg_kv_sender.clear()

Reproduction

  1. Start a PD disaggregation deployment with mooncake transfer backend
  2. Set mem_fraction_static such that max_total_num_tokens is relatively small (e.g., 284928)
  3. Send a request with input length slightly exceeding max_req_input_len (e.g., 284937 tokens)
  4. Observe: prefill returns prompt_tokens=1, decode returns garbage content unrelated to the input

Environment

  • SGLang version: 0.5.11
  • Model: zai-org/GLM-5.2-FP8 (DeepSeek-V3 architecture, MLA attention, MoE)
  • Transfer backend: mooncake
  • dp_size=32, tp_size=32, ep_size=32
  • mem_fraction_static=0.75, page_size=64
  • kv_cache_dtype=fp8_e4m3
  • enable_hierarchical_cache=true, hicache_ratio=2, hicache_size=200
  • disaggregation_mode=prefill
  • disaggregation_transfer_backend=mooncake
  • load_balance_method=follow_bootstrap_room
  • chunked_prefill_size=131072
  • context_len=1048576
  • max_total_num_tokens=284928

Activity

  1. github-actions commented on Sep 5, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions