[RFC][EPD] Avoid redundant encoder work when Prefill can reuse KV or embeddings #55639

Motivation

In an E-first EPD deployment, Encoder work can start before the selected Prefill instance determines whether reusable KV or embeddings already satisfy the relevant multimodal input. This creates a potential mismatch between the work started on E and the work actually required by P.

This RFC discusses whether Prefill-side reuse can be used to avoid such redundant Encoder work, and whether the benefit is sufficient to justify the additional coordination.

Encoder embeddings and prefix KV represent reusable results at different stages of the computation:

Image -> Encoder -> Embedding -> LLM Prefill -> KV
                       ^                       ^
                 Encoder cache           Prefix KV cache

There are two ways P may already have what it needs:

• KV coverage. The effective reusable KV prefix covers the positions corresponding to a multimodal occurrence. P no longer needs an embedding for that occurrence, even if the embedding itself is no longer cached.
• Embedding reuse. The KV prefix does not cover the occurrence, but P has a valid reusable embedding locally or can load one directly from an external cache. P still needs the embedding and must perform the remaining Prefill computation, but a new Encoder forward is unnecessary.

Here, a reusable embedding means an Encoder output that P can use for the current input, either from its local encoder cache or from an external cache it can load without a new E-side delivery. A multimodal processor-cache or metadata-cache hit alone does not satisfy this dependency.

Reusing the same image does not by itself imply KV reuse. When P still needs an embedding and can obtain it only from E, an E-side embedding-cache hit remains useful reuse: E can provide the embedding without recomputing it.

Current behavior

In scheduler.py, the waiting-request path determines the reusable KV prefix before deciding which Encoder inputs need to be scheduled. _try_schedule_encoder_inputs() operates on the remaining computation window, so a multimodal occurrence that is already fully covered by reusable KV does not need to be encoded again. For occurrences that still need embeddings, local encoder-cache hits can satisfy the dependency, and configured EC connectors can provide external embeddings instead of requiring Encoder computation.

The KV lookup does not require rerunning the Encoder. _gen_mm_extra_hash_keys() and hash_block_tokens() incorporate multimodal identity and position information into prefix hashing. Input processing sufficient to establish those identifiers, positions, and model metadata is still required, but the expensive Encoder forward is not.

In the reference EPD proxy, forward_non_stream() and forward_stream() await fanout_encoder_primer() before forwarding the request to P/PD:

Monolithic:
  Input processing -> KV prefix lookup -> Reuse an embedding or encode if needed

Reference E-first EPD:
  Dispatch E -> E-side cache handling / encoding / publication
             -> Forward to P/PD -> KV and embedding reuse decisions

E-side cache hits may already avoid Encoder recomputation, and connector behavior may avoid some payload transfers. The remaining issue is that P's later reuse decisions cannot suppress Encoder work that has already been started.

This describes the referenced E-first orchestration path and should not be interpreted as a claim about every possible EPD deployment.

Concrete scenarios

Consider requests with stable multimodal identity. An earlier request uses E0 and P0: E0 produces an image embedding, and P0 consumes it to compute the corresponding KV. A later request returns to P0 but is assigned to E1, which has no reusable embedding locally or through its configured external cache.

Two states illustrate the optimization opportunity:

• P retains KV covering the image occurrence. P no longer needs the embedding, regardless of whether it still retains that embedding. E1 may nevertheless encode the image before P0 determines that KV already covers it.
• P's relevant KV has been evicted, but its embedding remains cached. P must perform the remaining Prefill computation, but it can reuse its embedding. E1 may nevertheless encode the image before P0 determines that no new embedding is needed from E. An embedding that P can load directly from an external cache can satisfy the same demand, with a cache-load cost.

The same states can occur with a single Encoder if its embedding is evicted while the relevant KV or embedding remains available to P. E's embedding cache, P's embedding cache, and P's KV cache do not have identical placement or eviction lifecycles.

The work required to satisfy an occurrence can be summarized as follows. P-side embedding availability includes both local reuse and directly loadable external embeddings; E-side availability includes the selected E's local and configured external caches.

Conceptually, the demand for an embedding from E for an occurrence m is:

needs_embedding_from_E(m)
    = !KV_covered_P(m) && !reusable_embedding_available_P(m)

Avoiding Encoder forward, embedding delivery, and E-side control operations are separate opportunities. Metadata generation and connector coordination may still involve E even when no embedding is needed from it. When an embedding is needed from E, E-side cache reuse determines whether a new Encoder forward is necessary.

Proposed direction and trade-offs

One direction worth exploring is to use effective Prefill KV coverage and reusable embedding availability to determine which multimodal inputs still need an embedding from E. The focus is on using existing reusable state to avoid unnecessary Encoder work.

Two implementation strategies appear worth considering:

Prefill-side demand feedback. Reuse the scheduler's KV and embedding-availability decisions to identify the occurrences that still need embeddings from E, and communicate that demand before unnecessary work is committed.

Opportunistic short-circuiting. Preserve the existing eager path, but skip work when a reliable Prefill-side reuse decision becomes available before that work starts. This may reduce the impact on requests that need new encoding, at the cost of capturing fewer optimization opportunities.

The relevant trade-off is:

Encoder computation, delivery, and control work avoided after existing cache reuse versus the coordination, cache lookup or loading, synchronization, and possible loss of E/P overlap introduced by making Encoder work depend on Prefill's demand.

A design that requires every request to wait for a Prefill-side decision could save work on reuse paths while increasing latency for requests that need new encoding. Reusing an external embedding can also incur a load cost. These costs need to be weighed against the actual work avoided, rather than assuming each reuse opportunity saves a full Encoder forward and transfer.

The RFC therefore does not prescribe a specific control-flow change or API at this stage.

Any implementation should satisfy two correctness requirements:

• Satisfied dependencies. Skipping new Encoder computation must be justified by effective KV coverage of the corresponding multimodal occurrence or by a valid reusable embedding that can satisfy its remaining input dependency. Partial coverage and multiple multimodal inputs must be handled correctly.
• State validity. The KV or embedding state supporting the decision must remain valid when the request consumes it, or the system must provide a safe fallback if that state becomes unavailable.

A further design goal is to avoid significant additional latency or synchronization on the path that requires new encoding.

Validation

The following cases may be useful for characterizing the behavior:

The main quantity to establish is how often P can satisfy a multimodal occurrence through KV or embedding reuse while E still performs avoidable work. Distinguishing KV reuse, P-side embedding reuse, and existing E-side cache hits would help attribute the work avoided.

Useful signals include effective KV coverage, embedding availability and location, E requests, actual Encoder forward execution, E-to-P payload transfer, external cache loads, and request-level latency or throughput. These help distinguish avoided computation from avoided delivery or control work.

Controlled cases can demonstrate that the states are reachable, while representative multi-turn workloads would help determine how frequently they occur in practice and whether avoiding the work provides measurable end-to-end benefit.

Related Work and Open Questions

The EPD tracker #52409 provides the broader context for ongoing EPD development.

Several related changes address adjacent parts of the pipeline:

• #52359 proposes an asynchronous EPD pipeline with metadata publication before Encoder forward. This may provide useful infrastructure for earlier coordination, but it does not define how Prefill-side KV or embedding reuse should determine Encoder demand.
• #52697 allows KV consumers to omit multimodal embedding tensors when the corresponding prefetched KV is already available. This relates to the KV-reuse case, but does not affect upstream Encoder admission.
• #54659 adds metadata-only Prefill features, allowing Prefill-side processing without carrying the full multimodal embedding tensor. This may simplify earlier Prefill-side decisions, but does not by itself connect reuse decisions to Encoder scheduling.

The remaining design questions are:

1. How frequently does redundant Encoder work occur in representative EPD workloads due to KV reuse and P-side embedding reuse, after existing E-side cache mechanisms are taken into account?
2. At which layer should effective KV coverage and reusable embedding availability be converted into a decision about what P still needs from E?
3. Can such a mechanism avoid significant additional synchronization or latency for requests that need new encoding?

Answering these questions would help determine whether this optimization is worthwhile and what level of change to the EPD orchestration path is justified.