[AMD] Tune Kimi-K2.5 MXFP4 MI355X vLLM toward ATOM parity / 调优 Kimi-K2.5 MXFP4 MI355X vLLM，缩小与 ATOM 的性能差距

[Fangzhou-Ai]

Summary

• Apply the accuracy-gated candidate settings for amd/Kimi-K2.5-MXFP4 on MI355X with vLLM TP4.
• Keep image: TBD intentionally. This draft must not receive a sweep label until an official upstream vLLM ROCm image includes the required AITER MXFP4 backend.
• Preserve the existing TP4 and TP8 search spaces and extend only conc-end from 64 to 128 for both fixed-sequence workloads. Local evidence covers TP4 1k/1k at concurrency 4, 16, 32, and 128; TP8, the remaining points, and 8k/1k remain pending the official-image sweep.
• This changes benchmark configuration only; it does not patch vLLM or AITER source code.
• This draft replaces closed [Superseded] Kimi-K2.5 MI355X tuning draft / [已替代] Kimi-K2.5 MI355X 调优草稿 #2212 after cleaning internal experiment labels from the branch and PR metadata.

Effective settings

VLLM_ROCM_USE_AITER=1
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
VLLM_ROCM_USE_SKINNY_GEMM=0
VLLM_ROCM_USE_AITER_RMSNORM=0  # TP4
AITER_MXFP4_INTERMEDIATE=1
AITER_BYPASS_TUNE_CONFIG=0
AITER_MOE_SORT_BACKEND=auto
OMP_NUM_THREADS=1

--kv-cache-dtype fp8
--block-size 16
--max-num-batched-tokens 16384
--max-num-seqs 512
--async-scheduling
--no-enable-prefix-caching

The recipe also preserves the existing 0.90 GPU-memory utilization, MEC firmware workaround, trusted remote code, and multimodal encoder TP mode. It enforces a minimum server context of 9472 tokens so the 1k/1k configuration matches the locally validated server.

Local validation

These results validate local vLLM 94c0ef300 with AITER b2dd7703d, not the pending image:

1k/1k TP4 concurrency	Candidate total tok/s/GPU	ATOM total tok/s/GPU	Candidate / ATOM
4	227.71	246.47	92.39%
16	591.09	640.77	92.25%
32	843.05	918.26	91.81%
128	1736.45	1833.73	94.69%

• GSM8K: 1319/1319 requests, 97.35% strict and flexible exact match.
• bash -n benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_mi355x.sh
• Targeted matrix generation: 24 configurations, TP {4,8} × concurrency {4,8,16,32,64,128} × {1k/1k,8k/1k}.

AITER requirement

The tested kernel path requires:

1. ROCm/aiter e7731930 / #3753: transitional FlyDSL 0.2.1 bump.
2. ROCm/aiter 26d40611 / #3764: FlyDSL 0.2.2 compatibility and pin.
3. ROCm/aiter b2dd7703d / #3832: gfx950 RadeonFlow MXFP4 MoE backend and Kimi tuning.

Pinning b2dd7703d or a released descendant is sufficient because it already contains the first two commits. AITER v0.1.17-rc0 contains FlyDSL 0.2.2 but predates #3832, so it cannot reproduce the tested GEMM1/GEMM2 path.

Relationship to #2203

This is a materially different, measured alternative to #2203; only one should merge. #2203 copies K2.6-derived settings such as block size 1, 65,536 batched tokens, AITER RMSNorm, and custom MLA/compilation choices. This draft carries the locally validated Kimi-K2.5 settings that passed throughput scaling and full GSM8K evaluation.

Draft blockers

• Publish and insert an official upstream vLLM ROCm image containing AITER #3832 or a validated successor.
• Finish/update the matching upstream vllm-project/recipes documentation required by InferenceX policy.
• Run full-sweep-fail-fast, including evals, on the final image.
• Coordinate final disposition with Update kimik2.5 fp4 mi355x vLLM config #2203.

AI assistance was used to prepare the benchmark configuration and PR text. The submitter must review and defend every changed line.

中文说明

• 为 MI355X 上的 amd/Kimi-K2.5-MXFP4 vLLM TP4 基准测试应用已通过精度验证的候选配置。
• 有意保留 image: TBD。在上游正式发布包含所需 AITER MXFP4 后端的 vLLM ROCm 镜像之前，本草稿 PR 不应添加扫描标签。
• 保留现有 TP4 和 TP8 搜索空间，仅将两个固定序列长度负载的 conc-end 从 64 扩展到 128。本地数据覆盖 TP4 1k/1k 的并发点 4、16、32 和 128；TP8、其余并发点与 8k/1k 需要在正式镜像发布后通过 CI 扫描验证。
• 本 PR 只修改基准测试配置，不修改 vLLM 或 AITER 源码。
• 本草稿用于替代已关闭的 [Superseded] Kimi-K2.5 MI355X tuning draft / [已替代] Kimi-K2.5 MI355X 调优草稿 #2212，并清理了分支及 PR 元数据中的内部实验标签。

“Effective settings”代码块列出了完整启动配置。该配置还保留现有的 0.90 GPU 显存利用率、MEC 固件规避措施、trust-remote-code 和多模态编码器 TP 模式，并将服务端最小上下文长度设为 9472，以对齐本地验证的 1k/1k 配置。

本地验证

上表数据来自本地 vLLM 94c0ef300 与 AITER b2dd7703d，不代表待发布镜像的结果：候选配置在并发 4、16、32、128 下达到 ATOM 吞吐量的 91.81%–94.69%。GSM8K 共完成 1319/1319 个请求，严格和宽松 exact match 均为 97.35%。Shell 语法检查通过；定向矩阵生成了 24 个配置，即 TP {4,8} × 并发 {4,8,16,32,64,128} × {1k/1k,8k/1k}。

AITER 依赖

所测内核路径需要 ROCm/aiter e7731930（#3753，过渡到 FlyDSL 0.2.1）、26d40611（#3764，兼容并固定 FlyDSL 0.2.2）以及 b2dd7703d（#3832，为 gfx950 增加 RadeonFlow MXFP4 MoE 后端和 Kimi 调优）。直接使用 b2dd7703d 或包含它的后续正式版本即可，因为该提交已经包含前两项。

AITER v0.1.17-rc0 虽然包含 FlyDSL 0.2.2，但发布时间早于 #3832，因此无法复现本次验证所用的 GEMM1/GEMM2 路径。

与 #2203 的关系

本 PR 是与 #2203 实质不同、具有实测依据的替代方案，两者最终只能合并一个。#2203 复用了 K2.6 配置，包括 block size 1、65,536 batched tokens、AITER RMSNorm 以及自定义 MLA/编译选项；本草稿采用已完成吞吐量扩展测试和完整 GSM8K 评估的 Kimi-K2.5 候选配置。

草稿阻塞项

• 发布并填入包含 AITER #3832 或已验证后续版本的上游正式 vLLM ROCm 镜像。
• 按 InferenceX 要求完成或更新对应的上游 vllm-project/recipes 文档。
• 使用最终镜像运行包含精度评估的 full-sweep-fail-fast。
• 与 Update kimik2.5 fp4 mi355x vLLM config #2203 协调最终处理方式。

本 PR 的基准配置和说明文本使用了 AI 辅助。提交者必须逐行审核，并能够完整解释和维护这些改动。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[Fangzhou-Ai]

@chunfangamd need new aiter before proceeding