Repository navigation
[AMD] Add Kimi-K2.7-Code-MXFP4 to cookbook - #36246
Merged
1am9trash merged 7 commits intoAug 25, 2026
Merged
Conversation
Document the MXFP4 checkpoint (moonshotai/Kimi-K2.7-Code-MXFP4) validated on AMD MI350X/MI355X: add an INT4/MXFP4 Quantization axis to the deployment generator (MXFP4 gated to MI350X/MI355X), an Available Models entry, a configuration tip, and an MXFP4 deployment command example. Co-Authored-By: Claude <noreply@anthropic.com>
The MXFP4 checkpoint is published under the amd org, not moonshotai. Co-Authored-By: Claude <noreply@anthropic.com>
Emmanuel0612
requested review from
JustinTong0323,
sogalin,
wisclmy0611 and
zijiexia
as code owners
August 25, 2026 02:13
github-actions
Bot
added
the
documentation
Document the rocm/sgl-dev:v0.5.16-rocm720-mi35x-20260804 image used to validate the MXFP4 checkpoint on MI350X/MI355X. Co-Authored-By: Claude <noreply@anthropic.com>
Match launch_cookbook.sh: full AITER/ROCm env block, --attention-backend aiter, --mem-fraction-static 0.90, --disable-radix-cache, --enable-aiter-allreduce-fusion. Parser/DP-attention controls hide for MXFP4 (parsers auto-detect). Update the MDX example and config tip to the v0.5.17-rocm720-mi35x-20260812 image and matching command. Co-Authored-By: Claude <noreply@anthropic.com>
Add GSM8K accuracy (parallel 100/300/600/1200) and bench_serving latency results (concurrency 4-64) measured on 4x MI355X, mirroring the Kimi-K2.6 benchmark section layout. Keep the official model-card table as a reference. Co-Authored-By: Claude <noreply@anthropic.com>
5.1 Model Card (Official), then 5.2 MXFP4 Benchmark with 5.2.1 Accuracy and 5.2.2 Latency under a shared test environment. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Emmanuel0612
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
We enabled the Kimi-K2.7-Code-MXFP4 checkpoint in SGLang and tested it on AMD MI350X/MI355X GPUs. The cookbook page for Kimi-K2.7-Code only showed the INT4 checkpoint, so users had no easy way to find or launch the MXFP4 version. This PR adds the MXFP4 checkpoint to the cookbook so people can deploy it with a ready-to-use command.
Modifications
This is a documentation-only change. Two files are updated:
docs/src/snippets/autoregressive/kimi-k27-code-deployment.jsx(the interactive command generator)--model-path amd/Kimi-K2.7-Code-MXFP4and--mem-fraction-static 0.765. The INT4 command is unchanged.docs/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code.mdx(the page text)--quantizationflag needed), uses FP8 KV cache, and validated with therocm/sgl-dev:v0.5.16-rocm720-mi35x-20260804image.The launch settings come from a real, validated MI355X server run:
--tp 4,--kv-cache-dtype fp8_e4m3,--mem-fraction-static 0.765, ROCm aiter backend, with the MXFP4 format auto-detected from the checkpoint.Verifications
Checklist
CI States
Latest PR Test (Base): ✅ Run #32820019880
Latest PR Test (Extra): ❌ Run #32831875738
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.