Welcome to Mooncake

Welcome to Mooncake#

A KVCache-centric Disaggregated Architecture for LLM Serving.

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. Now both the Transfer Engine and Mooncake Store are open-sourced! This repository also hosts its technical report and the open-sourced traces.

🎉 Overview

Mooncake features a KVCache-centric disaggregated architecture that separates prefill and decode clusters. It also leverages underutilized CPU, DRAM, and SSD resources in GPU clusters to build a disaggregated KVCache pool.

At the center of Mooncake is a KVCache-centric scheduler that balances effective throughput with latency-related Service Level Objectives (SLOs). In highly overloaded scenarios, Mooncake uses prediction-based early rejection to avoid wasting computation on requests that cannot meet their SLOs. Experiments show that Mooncake is especially effective for long-context workloads, achieving up to a 525% throughput increase in simulated scenarios while meeting SLOs. Under real workloads, Mooncake’s architecture enables Kimi to handle 75% more requests.

🔄 Updates

Oct 10, 2026: TensorRT-LLM merged a native Mooncake Store connector into KVCMv2, enabling content-addressed KV cache pooling and prefix reuse across serving instances.
Oct 6, 2026: TokenSpeed v0.1.1 includes Mooncake Store as an L3 KV cache backend beneath its host cache, alongside Mooncake Transfer Engine support for PD and EPD disaggregation.
Sept 17, 2026: vLLM added cross-encoder embedding reuse through Mooncake Store, complementing its encoder-cache transfer connector for multimodal disaggregated serving. Both capabilities are included in vLLM v0.31.0.
Aug 28, 2026: LMDeploy merged Mooncake Store support, enabling shared KV cache pooling and prefix reuse across PyTorch inference instances. Included in LMDeploy v0.18.0.
Aug 24, 2026: AgentX / InferenceX v3 highlights Mooncake’s contributions to agentic inference workloads, including ROCm wheels, CI, and packaging improvements. Mooncake optimization details.

More

Aug 20, 2026: Mooncake is integrated into Miles as a rollout data-transfer backend for the fragmented, heterogeneous data moving between rollout and training in disaggregated RL. Blogs: KVCache.AI, lmsys.
Aug 17, 2026: Mooncake is integrated into Speculators as a distributed backend for multi-node online training, efficiently moves hidden-state between vLLM inference workers and Speculators trainers through RDMA, eliminating the need for massive hidden-state storage in offline training. Benchmark on GB300 NVL72.
Aug 5, 2026: llm-d added a Mooncake PD deployment guide, complementing its Mooncake Store tiered offloading integration for vLLM serving on Kubernetes.
Aug 1, 2026: Dynamo merged an event-driven Mooncake shared-cache index for SGLang HiCache routing, using Mooncake object events to track reusable KV pages in the shared pool.
Jul 24, 2026: FlexKV added Mooncake Store as a distributed REMOTE-tier backend, enabling content-addressed KV cache sharing alongside its existing Mooncake Transfer Engine integration.
Jul 12, 2026: TransferQueue added GPUDirect RDMA support for its Mooncake Store backend, enabling GPU clients to exchange RL tensors with the host-memory store through GDR.
May 7, 2026: 🚀 vLLM officially features Mooncake Store — a deep dive into how Mooncake’s distributed KVCache engine supercharges vLLM inference with high-throughput, memory-efficient, cross-instance KV cache sharing!
Apr 29, 2026: SGLang introduces RDMA-based P2P weight transfer for large-scale distributed RL using Mooncake TransferEngine, achieving 7x faster weight updates for the 1T-parameter Kimi-K2 model (53s → 7.2s) with zero-copy RDMA transfer across thousands of GPUs.
Apr 3, 2026: LMCache introduced a native Mooncake Store L2 adapter for multiprocess mode, connecting the LMCache MP server to the distributed KV cache pool.
Apr 1, 2026: The TENT technical report introduces declarative data-movement orchestration for heterogeneous interconnects, with evaluations in SGLang HiCache and checkpoint-engine workloads.
Mar 19, 2026: TorchSpec: Speculative Decoding Training at Scale is open sourced, using Mooncake to decouple inference and training via efficient hidden states management.
Mar 16, 2026: FastDeploy added Mooncake Store cache read/write support to its PD pipeline, complementing its global KV cache pooling across inference instances.
Mar 5, 2026: LightX2V now supports disaggregated deployment based on Mooncake, enabling encoder/transformer service decoupling with Mooncake Transfer Engine for high-performance cross-device and cross-machine data transfer. Details in blog.
Mar 3, 2026: verl added a Mooncake checkpoint-engine backend for peer-to-peer weight synchronization between training and rollout workers.
Feb 25, 2026: SGLang merged Encoder Global Cache Manager, introducing a Mooncake-powered global multimodal embedding cache that enables cross-instance sharing of ViT embeddings to avoid redundant GPU computation.
Feb 24, 2026: vLLM-Omni introduces disaggregated inference connectors with support for both MooncakeStoreConnector and MooncakeTransferEngineConnector for multi-node omni-modality pipelines.
Feb 12, 2026: Mooncake Joins PyTorch Ecosystem We are thrilled to announce that Mooncake has officially joined the PyTorch Ecosystem!
Jan 28, 2026: FlexKV, a distributed KV store and cache system from Tencent and NVIDIA in collaboration with the community, now supports distributed KVCache reuse with the Mooncake Transfer Engine.
Dec 27, 2025: Collaboration with ROLL! Check out the paper here.
Dec 23, 2025: SGLang introduces Encode-Prefill-Decode (EPD) Disaggregation with Mooncake as a transfer backend. This integration allows decoupling compute-intensive multimodal encoders (e.g., Vision Transformers) from language model nodes, utilizing Mooncake’s RDMA engine for zero-copy transfer of large multimodal embeddings.
Dec 19, 2025: Mooncake Transfer Engine has been integrated into TensorRT LLM for KVCache transfer in PD-disaggregated inference.
Dec 19, 2025: Mooncake Transfer Engine has been directly integrated into vLLM v1 as a KV Connector in PD-disaggregated setups.
Nov 07, 2025: RBG + SGLang HiCache + Mooncake, a role-based out-of-the-box solution for cloud native deployment, which is elastic, scalable, and high-performance.
Oct 15, 2025: SGLang introduced Mooncake EP and a collective communication backend as the foundation for fault-tolerant Elastic Expert Parallel inference.
Sept 18, 2025: Mooncake Store empowers vLLM Ascend by serving as the distributed KV cache pool backend.
Sept 10, 2025: SGLang officially supports Mooncake Store as a hierarchical KV caching storage backend. The integration extends RadixAttention with multi-tier KV cache storage across device, host, and remote storage layers.
Sept 10, 2025: The official & high-performance version of Mooncake P2P Store is open-sourced as checkpoint-engine. It has been successfully applied in K1.5 and K2 production training, updating Kimi-K2 model (1T parameters) across thousands of GPUs in ~20s.
Aug 23, 2025: xLLM high-performance inference engine builds hybrid KV cache management based on Mooncake, supporting global KV cache management with intelligent offloading and prefetching.
Aug 18, 2025: vLLM-Ascend integrates Mooncake Transfer Engine for KV cache register and disaggregate prefill, enabling efficient distributed inference on Ascend NPUs.
Jul 20, 2025: Mooncake powers the deployment of Kimi K2 on 128 H200 GPUs with PD disaggregation and large-scale expert parallelism, achieving 224k tokens/sec prefill throughput and 288k tokens/sec decode throughput.
Jun 20, 2025: Mooncake becomes a PD disaggregation backend for LMDeploy.
May 9, 2025: NIXL officially supports Mooncake Transfer Engine as a backend plugin.
May 8, 2025: Mooncake x LMCache unite to pioneer KVCache-centric LLM serving system.
May 5, 2025: Supported by Mooncake Team, SGLang release guidance to deploy DeepSeek with PD Disaggregation on 96 H100 GPUs.
Apr 22, 2025: LMCache officially supports Mooncake Store as a remote connector.
Apr 10, 2025: SGLang officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
Mar 7, 2025: We open-sourced the Mooncake Store, a distributed KVCache based on Transfer Engine. vLLM’s xPyD disaggregated prefilling & decoding based on Mooncake Store will be released soon.
Feb 25, 2025: Mooncake receives the Best Paper Award at FAST 2025!
Feb 21, 2025: The updated traces used in our FAST’25 paper have been released.
Dec 16, 2024: vLLM officially supports Mooncake Transfer Engine for disaggregated prefilling and KV cache transfer.
Nov 28, 2024: We open-sourced the Transfer Engine, the central component of Mooncake. We also provide two demonstrations of Transfer Engine: a P2P Store and vLLM integration.
July 9, 2024: We open-sourced the trace as a JSONL file.
June 27, 2024: We present a series of Chinese blogs with more discussions on zhihu 1, 2, 3, 4, 5, 6, 7.
June 26, 2024: Initial technical report release.