Skip to content

[DSv4] Refresh GB200 dynamo-vLLM 1k1k sweep#2190

Open
Oseltamivir wants to merge 1 commit into
mainfrom
dsv4-gb200-restore-1k1k
Open

[DSv4] Refresh GB200 dynamo-vLLM 1k1k sweep#2190
Oseltamivir wants to merge 1 commit into
mainfrom
dsv4-gb200-restore-1k1k

Conversation

@Oseltamivir

Copy link
Copy Markdown
Collaborator

Summary

Second PR of the DSv4 submission refresh pass (first: #2189). Fixes the unusable GB200 dsv4 1k1k data.

Problem: the site's GB200 dsv4 1k1k shows 9 rows, all from 2026-04-25, one dynamo-vllm disagg config at small GPU counts — 217 out tok/s/GPU at 30 tok/s/user vs GB300's 9,050. Root cause: the day-0 1k1k recipes (#1129, 2026-04-24) were deleted two days later by the deepseek-v4-pro-sa curation (5b0347f43, "only 8k/1k is in scope"), so every GB200 dsv4 config has been 8k1k-only since — the April day-0 rows were frozen with no way to refresh them.

Change:

  • Re-add the three day-0 1k1k topologies to dsv4-fp4-gb200-dynamo-vllm with engine tuning unchanged:
    • 1p1d-dep8-tep8 — conc 1, 4, 8, 16, 32, 64 (5 nodes incl. infra)
    • 1p1d-dep8-dep16 — conc 128, 256, 1024, 2048, 4096 (7 nodes)
    • 3p1d-dep8-dep16 — conc 4096, 8192 (11 nodes, same ceiling as the existing 8k1k low-middle-curve)
  • Restore the recipe YAMLs under benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/1k1k/, re-based on the current 8k1k -sa layout: model.container = vllm/vllm-openai:v0.20.0-ubuntu2404 (matches the config image:/alias key per RECIPES.md), dynamo.wheel 1.2.0.dev20260426, dedicated NATS/etcd infra node. Recipe benchmark.concurrencies match the master conc-lists exactly.

The config image is intentionally left at v0.20.0-ubuntu2404 — it's shared with the five working 8k1k scenarios, and the restored topologies were day-0-validated; bumping the whole GB200 vLLM stack is out of scope here.

Validation: generate_sweep_configs.py --model-prefix dsv4 --framework dynamo-vllm --runner-type gb200 produces the expected 12 configs (3 restored 1k1k rows + existing 8k1k); validate_perf_changelog.py --base-ref main passes; recipe YAMLs parse.

中文说明

DSv4 基准数据刷新批次的第二个 PR(第一个:#2189),修复不可用的 GB200 dsv4 1k1k 数据。

问题: 官网 GB200 dsv4 1k1k 仅有 9 行数据,全部来自 2026-04-25 的单一 dynamo-vllm 分离式配置(GPU 数量较小)——在 30 tok/s/user 交互性下仅 217 out tok/s/GPU,而 GB300 为 9,050。根本原因:day-0 的 1k1k 配方(#1129,2026-04-24)在两天后被 deepseek-v4-pro-sa 整理提交(5b0347f43,"only 8k/1k is in scope")删除,此后所有 GB200 dsv4 配置均只有 8k1k——4 月的 day-0 数据被冻结,无法刷新。

改动:

  • dsv4-fp4-gb200-dynamo-vllm 重新加入三个 day-0 1k1k 拓扑(引擎调优不变):
    • 1p1d-dep8-tep8 — conc 1, 4, 8, 16, 32, 64(含基础设施节点共 5 节点)
    • 1p1d-dep8-dep16 — conc 128, 256, 1024, 2048, 4096(7 节点)
    • 3p1d-dep8-dep16 — conc 4096, 8192(11 节点,与现有 8k1k low-middle-curve 上限一致)
  • 恢复 benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/1k1k/ 下的配方 YAML,并对齐当前 8k1k -sa 布局:model.container = vllm/vllm-openai:v0.20.0-ubuntu2404(按 RECIPES.md 要求与配置的 image: 别名键一致)、dynamo.wheel 1.2.0.dev20260426、专用 NATS/etcd 基础设施节点。配方内 benchmark.concurrencies 与主配置的 conc-list 完全一致。

配置镜像有意保持 v0.20.0-ubuntu2404——它与五个正常运行的 8k1k 场景共用,且恢复的拓扑经过 day-0 验证;整体升级 GB200 vLLM 栈不在本 PR 范围内。

验证:generate_sweep_configs.py --model-prefix dsv4 --framework dynamo-vllm --runner-type gb200 生成预期的 12 个配置(3 个恢复的 1k1k + 现有 8k1k);validate_perf_changelog.py --base-ref main 通过;配方 YAML 解析正常。

The day-0 1k1k recipes (#1129) were dropped by the deepseek-v4-pro-sa
curation (5b0347f), freezing GB200 dsv4 1k1k submissions at the
2026-04-25 day-0 run. Re-add the three day-0 topologies (1p1d-dep8-tep8,
1p1d-dep8-dep16, 3p1d-dep8-dep16) with engine tuning unchanged; recipe
container/dynamo/infra fields re-based on the current 8k1k -sa set.

中文:恢复 GB200 dsv4 dynamo-vLLM 的 1k1k 扫描。day-0 的 1k1k 配方在
deepseek-v4-pro-sa 整理时被移除,导致官网 GB200 1k1k 数据停留在
2026-04-25。重新加入三个 day-0 拓扑(引擎调优不变),配方的
container/dynamo/infra 字段与当前 8k1k -sa 集合对齐。
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Comment on lines +130 to +149
kv-cache-dtype: "fp8"
tensor-parallel-size: 8
pipeline-parallel-size: 1
enable-expert-parallel: true
max-model-len: 3072
max-num-seqs: 64
max-cudagraph-capture-size: 64
max-num-batched-tokens: 64
trust-remote-code: true
no-enable-prefix-caching: true
block-size: 256
attention-config: '{"use_fp4_indexer_cache":true}'
compilation-config: '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","pass_config":{"fuse_allreduce_rms":false}}'
gpu-memory-utilization: 0.9
stream-interval: 50
no-disable-hybrid-kv-cache-manager: true
enable-sleep-mode: true
tokenizer-mode: deepseek_v4

benchmark:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 In the restored disagg-gb200-1p1d-dep8-tep8.yaml recipe, the decode block sets tensor-parallel-size: 8 and enable-expert-parallel: true (no data-parallel-size), so the decode engine actually runs with expert parallelism enabled. But configs/nvidia-master.yaml labels this decode as tp: 8, ep: 1, dp-attn: false (pure TEP), so the published conc 1-64 row for this topology will be mislabeled relative to what the engine actually runs. The fix is to comment out enable-expert-parallel in the decode block, matching the 8k1k disagg-gb200-low-latency.yaml sibling this recipe is explicitly based on.

Extended reasoning...

The bug: disagg-gb200-1p1d-dep8-tep8.yaml's decode block (lines 130-149) sets tensor-parallel-size: 8, enable-expert-parallel: true, and no data-parallel-size. In vLLM, EP world size = tp × dp, so with tp=8/dp=1 this actually shards MoE experts across the 8 TP ranks — a real, different execution mode from plain tensor-parallel decode with replicated experts. Meanwhile configs/nvidia-master.yaml's new search-space entry for this exact recipe (the conc-list: [1, 4, 8, 16, 32, 64] block referencing this CONFIG_FILE) labels decode as tp: 8, ep: 1, dp-attn: false — the pure-TEP convention.\n\nWhy the label matters: per AGENTS.md's srt-slurm section, srtctl reads only the recipe to actually launch the engine, while generate_sweep_configs.py reads nvidia-master.yaml for the frontend labels shown on the published site. These two are decoupled by design, and AGENTS.md explicitly warns that 'Recipe-only edits mislabel results.' That's exactly what happens here: the engine runs with EP enabled, but the site will report/plot the row as ep=1 pure-TEP.\n\nConfirmed against the established convention: I checked the analogous 8k1k sibling this recipe's own header comment claims to mirror ('Local deltas vs upstream 8k/1k sibling: same as the 8k/1k recipe'). 8k1k/disagg-gb200-low-latency.yaml's decode block (lines 112-133) has the identical topology (tp:8, no dp) but explicitly comments out enable-expert-parallel and data-parallel-size:\n\n tensor-parallel-size: 8\n pipeline-parallel-size: 1\n# data-parallel-size: 8\n# data-parallel-rpc-port: 13345\n# enable-expert-parallel: true\n\nAnd master's entry for that 8k1k sibling (lines 8741-8752) carries the same ep: 1, dp-attn: false decode label. So the convention across every GB200 dsv4 decode in this file is: ep:1/dp-attn:false label ⇔ enable-expert-parallel commented out in the recipe; ep:8+/dp-attn:true label ⇔ it's active with a matching data-parallel-size. The new 1k1k recipe is the only pure-TEP-labeled decode that leaves enable-expert-parallel: true active.\n\nStep-by-step proof:\n1. Master config (new entry, this PR): conc-list: [1,4,8,16,32,64] → decode tp:8, ep:1, dp-attn:false, referencing CONFIG_FILE=recipes/vllm/deepseek-v4/1k1k/disagg-gb200-1p1d-dep8-tep8.yaml.\n2. That recipe's decode block: tensor-parallel-size: 8, enable-expert-parallel: true, no data-parallel-size → vLLM launches with EP world size 8 (experts sharded across the 8 TP ranks), not plain TP.\n3. Compare to the 8k1k sibling with the same master label (ep:1, dp-attn:false): its recipe comments out enable-expert-parallel/data-parallel-size, so it truly runs plain TP=8 with replicated experts.\n4. Result: identical master label (ep:1) maps to two different actual decode execution modes — one is really EP=8, the other is really EP disabled. The 1k1k row's reported topology doesn't match what produced its numbers.\n\nFix: comment out (or delete) enable-expert-parallel: true in the decode block of disagg-gb200-1p1d-dep8-tep8.yaml (1k1k) to match the 8k1k sibling and the ep:1 label — consistent with the PR's own stated intent of re-basing this recipe on the current 8k1k -sa layout. This is a valid vLLM config either way (it won't crash CI), so it's a data-correctness/mislabeling issue rather than a functional break.

Comment on lines +87 to +106
prefill:
kv-transfer-config: '{"kv_connector": "NixlConnector", "kv_role": "kv_both"}'
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
kv-cache-dtype: "fp8"
tensor-parallel-size: 1
pipeline-parallel-size: 1
data-parallel-size: 8
data-parallel-rpc-port: 13345
enable-expert-parallel: true
enforce-eager: true
max-model-len: 3072
max-num-seqs: 16
max-num-batched-tokens: 16384
trust-remote-code: true
no-enable-prefix-caching: true
no-enable-flashinfer-autotune: true
block-size: 256
gpu-memory-utilization: 0.88
no-disable-hybrid-kv-cache-manager: true
enable-sleep-mode: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Both new dep8-dep16 recipes (1p1d and 3p1d) omit tokenizer-mode: deepseek_v4 in their prefill and decode vllm_config blocks, unlike every other DeepSeek-V4-Pro recipe in the tree (including the tep8 sibling added in this same PR).

Extended reasoning...

What's wrong

disagg-gb200-1p1d-dep8-dep16.yaml and disagg-gb200-3p1d-dep8-dep16.yaml set no tokenizer-mode anywhere in their vllm_config.prefill or vllm_config.decode blocks. Every other DeepSeek-V4-Pro vLLM recipe in benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/ — all 8k1k GB200/B200/GB300 recipes, the agentic recipes, and critically the disagg-gb200-1p1d-dep8-tep8.yaml sibling added in this same PR (lines 125 and 147) — sets tokenizer-mode: deepseek_v4 in both the prefill and decode sections.

Why this happened

The header comment on the affected file states it was "Extrapolated from kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml adjusted for DSV4-Pro's DP>=8 minimum." Kimi-K2.5 is a different model that does not require the DeepSeek-specific tokenizer, so the extrapolation carried over everything except this one model-specific field. By contrast, the tep8 sibling recipe in the same PR was mirrored directly from NVIDIA's 8k1k DSV4 reference recipe, which is why it correctly retained tokenizer-mode: deepseek_v4.

Why nothing else catches it

tokenizer-mode is not injected by the config/recipe generation pipeline (generate_sweep_configs.py) — it is a raw pass-through field into the vLLM launch args from the recipe YAML itself. There's no default-injection or validation step downstream that would backfill it, and deepseek_v4 is a genuinely distinct, non-auto-selected tokenizer mode (see utils/bench_serving/benchmark_serving.py around line 1271, where it is listed as a separate choice alongside auto/slow/mistral/custom, with a comment noting the stock HF AutoTokenizer may not recognize deepseek_v4 and vLLM's own loader must be used instead).

Impact

Without this setting, vLLM falls back to its default (auto) tokenizer resolution for DeepSeek-V4-Pro, which will either fail to load a workable tokenizer at engine startup or silently mis-tokenize inputs/outputs. These two recipes back the conc-list: [128, 256, 1024, 2048, 4096] and conc-list: [4096, 8192] scenarios in configs/nvidia-master.yaml — i.e. most of the 1k1k sweep this PR is restoring. The result would either be failed CI runs or (worse) benchmark numbers published for an incorrectly-tokenized model, defeating the stated purpose of the PR.

Proof by comparison

  1. Grep the tree for tokenizer-mode: deepseek_v4 — it appears twice (prefill + decode) in every existing DSv4-Pro recipe, and twice in the new disagg-gb200-1p1d-dep8-tep8.yaml (lines 125, 147) added in this PR.
  2. The same grep against disagg-gb200-1p1d-dep8-dep16.yaml (lines 87-106 shown, prefill/decode vllm_config blocks) and disagg-gb200-3p1d-dep8-dep16.yaml returns zero matches.
  3. The fix is a one-line addition per section: add tokenizer-mode: deepseek_v4 to both the prefill: and decode: entries of vllm_config in both files, mirroring the tep8 recipe and every 8k1k recipe.

Comment on lines +10 to +12
#
# Local deltas vs upstream 8k/1k sibling: same as the 8k/1k recipe — see
# ../8k1k/disagg-gb200-1p1d-dep8-tep8.yaml for the full deviation list.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Two header-comment path references in the restored 1k1k recipes point to nonexistent files. In disagg-gb200-1p1d-dep8-tep8.yaml (lines 10-12), the comment points to '../8k1k/disagg-gb200-1p1d-dep8-tep8.yaml' but the actual GB200 8k1k low-latency sibling is named disagg-gb200-low-latency.yaml. In disagg-gb200-1p1d-dep8-dep16.yaml (lines 2-4), the comment references 'kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml' but the actual directory is kimi-k2.5-fp4/1k1k/ (missing the -fp4 suffix). Both are comment-only with no runtime impact; worth a quick fix so future readers can actually follow the references.

Extended reasoning...

What the bug is: Two of the three newly-restored 1k1k recipe YAMLs contain header comments that cross-reference sibling/precedent files using paths that do not exist in this repository.

  1. In disagg-gb200-1p1d-dep8-tep8.yaml (lines 10-12), the comment reads: "Local deltas vs upstream 8k/1k sibling: same as the 8k/1k recipe — see ../8k1k/disagg-gb200-1p1d-dep8-tep8.yaml for the full deviation list." There is no file named disagg-gb200-1p1d-dep8-tep8.yaml under benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/. That directory names its GB200 recipes descriptively (e.g. disagg-gb200-low-latency.yaml, disagg-gb200-low-middle-curve.yaml, disagg-gb200-high-tpt-megamoe.yaml) rather than by topology string. The actual 8k1k sibling implementing the same 1-prefill-DP8 / 1-decode-TP8 low-latency topology is disagg-gb200-low-latency.yaml.

  2. In disagg-gb200-1p1d-dep8-dep16.yaml (top-of-file comment), the precedent this recipe was extrapolated from is written as kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml. The only kimi directory that exists under vllm/ is kimi-k2.5-fp4/ (with the -fp4 suffix) — the referenced filename itself is correct and does exist, but at vllm/kimi-k2.5-fp4/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml, not the path written in the comment.

Code path / why nothing catches this: These are plain YAML comments (lines starting with #), never parsed by generate_sweep_configs.py, validate_perf_changelog.py, or any recipe loader — they're documentation for humans skimming the file. Nothing in the PR's stated validation (config generation, changelog validation, YAML parse) would ever touch these strings, so a broken reference here can't be caught by CI; it only surfaces when a person manually tries to open the referenced file.

Impact: Zero runtime impact — the sweep, config generation, and benchmark execution are all unaffected. The only cost is to a future engineer who reads the comment, tries to open the referenced file to check 'the full deviation list' or find the extrapolation precedent, and hits a dead path. For (1) this is a real dead-end since the correct file has a completely different name (disagg-gb200-low-latency.yaml vs. the referenced disagg-gb200-1p1d-dep8-tep8.yaml) and wouldn't be trivially guessable. For (2) the impact is milder — the filename itself is correct and unique in the repo, so a reader who greps for the filename (rather than manually cd'ing into the written path) finds it immediately; a verifier flagged this as too minor to be worth a review comment on its own.

Proof (bug 1): ls benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/ shows no disagg-gb200-1p1d-dep8-tep8.yaml; the GB200 file implementing the same DP8-prefill/TP8-decode low-latency topology is disagg-gb200-low-latency.yaml (confirmed via its name: svf-vllm-disagg-gb200-low-latency and decode tp:8/ep:1 config in configs/nvidia-master.yaml). Following the comment's literal path therefore resolves to a missing file.

Proof (bug 2): ls benchmarks/multi_node/srt-slurm-recipes/vllm/ shows the only kimi directory is kimi-k2.5-fp4; find for disagg-gb200-1p1d-dep4-dep16.yaml locates it only at kimi-k2.5-fp4/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml. The comment's written path kimi-k2.5/1k1k/... (no -fp4) does not exist as a directory.

Fix: Repoint the first comment at ../8k1k/disagg-gb200-low-latency.yaml, and add the missing -fp4 to the second comment's directory (kimi-k2.5-fp4/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml). Both are one-line comment edits with no functional change.

On the refutation: One verifier argued bug 2 is too trivial to report because the filename is unique and easily grep-able despite the wrong parent directory. That's a fair point for bug 2 in isolation, but bug 1's broken reference is a harder dead-end (the correct file has an entirely different name, not a typo'd path), so the pair together is worth a single combined nit-level note — cheap for the author to fix in the same pass, not worth blocking merge over.

osl: 1024
concurrencies: "128x256x1024x2048x4096"
req_rate: "inf"
use_chat_template: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The three new 1k1k vLLM DSv4-Pro recipes set benchmark.use_chat_template: false with no custom_tokenizer, while every one of the 20 existing DeepSeek-V4-Pro vLLM 8k1k recipes uses use_chat_template: true + custom_tokenizer: sa_bench_tokenizers.vllm_deepseek_v4.VLLMDeepseekV4Tokenizer. This looks like a leftover from the kimi-k2.5 template cited in the header comments rather than a deliberate choice, and isn't called out in the PR description — worth a quick confirmation, though the actual measurement impact on this fixed-ISL/OSL sweep is likely small.

Extended reasoning...

All three new 1k1k recipes (disagg-gb200-1p1d-dep8-tep8.yaml, disagg-gb200-1p1d-dep8-dep16.yaml, disagg-gb200-3p1d-dep8-dep16.yaml) set benchmark.use_chat_template: false and omit custom_tokenizer entirely. Every one of the 20 existing DeepSeek-V4-Pro recipes under benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/ sets use_chat_template: true paired with custom_tokenizer: sa_bench_tokenizers.vllm_deepseek_v4.VLLMDeepseekV4Tokenizer, with zero exceptions. The header comments in the new files say they were extrapolated from kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml and NVIDIA's 8k1k recipe — neither of which uses the vLLM-DSv4 tokenizer/chat-template convention — which is consistent with this being a copy-paste leftover rather than an intentional change, and it is not mentioned anywhere in the PR description.

I verified the client-side code path in utils/bench_serving/benchmark_serving.py: use_chat_template gates whether _apply_chat_template() (and, when --dsv4 is set, the dedicated DeepSeek-V4 encoder in encoding_dsv4.py) is applied to generated prompts (lines 211-212, 334-335). There's also a hard guard, if dsv4 and not use_chat_template: raise ValueError(...), but I confirmed in benchmarks/benchmark_lib.sh (lines 430-432) that --dsv4 auto-enables --use-chat-template in the shell wrapper that builds the CLI invocation, so that ValueError cannot actually fire here — no crash risk, contrary to what a naive reading of the guard would suggest.

A verifier raised a solid refutation that I confirmed by grepping the repo: use_chat_template: false with no custom_tokenizer is not unique to these three new files — several existing sglang DeepSeek-V4-Pro 8k1k recipes (e.g. disagg-gb200-1p1d-dep8-dep16-6-c512.yaml, disagg-gb200-4p1d-dep8-dep16-12-c4096.yaml, disagg-gb200-2p1d-dep8-dep16-8-c1536.yaml) also set use_chat_template: false with no custom tokenizer field at all. So the "always true" convention is specific to the vLLM 8k1k DSv4 recipes (still 20/20 with no exceptions there, which is the directly comparable set for these new vLLM recipes), not a universal rule across every DSv4 recipe in the repo. This softens the "clear deviation" framing somewhat, though it doesn't fully explain away the discrepancy since these new files are vLLM recipes being compared against the vLLM convention.

On impact: since benchmark.type here is sa-bench with fixed isl/osl (synthetic random-token generation, not a real corpus), the chat template only changes how the ~1024 input tokens are wrapped/formatted, not the token countbenchmark_serving.py line 244-245 explicitly subtracts the chat-template overhead so the total input length still lands on the target ISL either way. Throughput/latency at a fixed ISL/OSL is driven primarily by topology and batch size, not by whether the tokens are wrapped in a chat template, so the "skews the whole sweep" framing in the original report is likely overstated for this fixed-seq-len benchmark type. The more defensible concern is data-comparability/methodology: the restored 1k1k rows would be generated by a different code path (default tokenizer, no DSv4 chat-template) than the comparable 8k1k rows for the same model, which is worth the author confirming is intentional before publishing to the site — but I couldn't find a concrete mechanism by which it changes the actual throughput/latency numbers with a fixed-ISL/OSL sweep.

Proof walkthrough: For disagg-gb200-1p1d-dep8-dep16.yaml, isl: 1024, osl: 1024. With use_chat_template: true (as in the 8k1k sibling recipes), benchmark_serving.py would apply _apply_chat_template(), compute chat_template_len, and generate 1024 - chat_template_len raw tokens so the templated prompt totals ~1024 tokens. With use_chat_template: false (as configured here), it skips that step and generates exactly 1024 raw tokens directly. In both cases the model receives ~1024 input tokens and produces up to 1024 output tokens — the throughput-determining quantities are the same; what differs is only the token content/formatting, plus which tokenizer encodes it (default HF tokenizer vs. the dedicated VLLMDeepseekV4Tokenizer). That difference matters for methodology consistency across the site's DSv4 rows, but not for whether the reported tok/s numbers for this recipe are individually valid.

Recommend the author confirm whether this was intentional (e.g., because these are day-0-derived recipes predating the -sa tokenizer convention) or fix it to match the 8k1k vLLM convention for consistency — but I don't think it rises to a blocking functional defect given the fixed-ISL/OSL benchmark type and the lack of a crash path.

@github-actions

Copy link
Copy Markdown
Contributor

@Oseltamivir Oseltamivir changed the title [DSv4] Restore GB200 dynamo-vLLM 1k1k sweep / 恢复 GB200 dynamo-vLLM 1k1k 扫描 [DSv4] Refresh GB200 dynamo-vLLM 1k1k sweep Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant