[AgentX] vLLM Deepseek-V4 B200/B300 agg

[cquil11]

Summary

Refresh the single-node agentic-coding vLLM recipes for DeepSeek-V4-Pro FP4
on B200 and B300.

Testing

• full-sweep CI green

Original PR: #2188 by @ivanium

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🟡 benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh:190-203 — The B200 DEP recipe (tp:8, ep:8, dp-attn:true) computes MAX_NUM_SEQS=$((2*CONC)) unconditionally (line 206), but under DP-attention each of the $TP DP ranks is an independent vLLM engine that only sees roughly CONC/TP of the total concurrency, so this over-provisions --max-num-seqs and --max-cudagraph-capture-size by a factor of TP per rank. The sibling dsv4_fp4_b300_vllm.sh added in this same PR divides by TP for the DP-attention case (MAX_NUM_SEQS=$((2*CONC/TP))), so the B200 script should mirror that branch for consistency with the intended DEP recipe.

  Extended reasoning...

  The bug: In dsv4_fp4_b200_vllm.sh, MAX_NUM_SEQS=$((2 * CONC)) (line 206) is computed unconditionally with no branch on DP_ATTENTION. This value is then used both for --max-num-seqs and, in the same PR's new addition, for --max-cudagraph-capture-size (line ~233).

  Under DP-attention (DP_ATTENTION=true), PARALLEL_ARGS is set to --tensor-parallel-size 1 --data-parallel-size $TP (lines 187-189), meaning each of the $TP GPUs runs an independent vLLM engine (a DP rank), and vllm-router fronts them with --intra-node-data-parallel-size $TP and consistent_hash policy so that AgentX session trees are distributed across ranks. Each rank therefore sees only roughly CONC/TP sessions in steady state, not the full CONC. --max-num-seqs and --max-cudagraph-capture-size are per-engine settings, so the correct per-rank cap under DP-attention is ~2*CONC/TP, not 2*CONC.

  Why this is inconsistent, not hypothetical: this exact issue is handled correctly in the sibling script touched by the same PR, dsv4_fp4_b300_vllm.sh. It computes:

  if [ "$DP_ATTENTION" = "true" ]; then
      # The DEP source recipe enforces 2*CONC = DP_WORLD_SIZE*MAX_NUM_SEQS.
      MAX_NUM_SEQS=$((2 * CONC / TP))
  else
      MAX_NUM_SEQS=$((2 * CONC))
  fi
  

  along with a startup guard requiring 2*CONC to be evenly divisible by TP. The B200 script never got the equivalent branch, so it keeps using the pre-DEP aggregate formula for every topology, including the DEP8 tiers.

  Step-by-step proof: Take the B200 DEP8 tier in nvidia-master.yaml (tp: 8, ep: 8, dp-attn: true) at conc: 72, one of the values in the new conc-list.

  1. TP=8, CONC=72, DP_ATTENTION=true.
  2. PARALLEL_ARGS becomes --tensor-parallel-size 1 --data-parallel-size 8 — 8 independent DP-rank engines.
  3. vllm-router load-balances the 72 concurrent session trees across those 8 ranks via consistent hashing, so each rank's steady-state load is ~9 sessions (~18 with the intended 2x headroom).
  4. But MAX_NUM_SEQS=$((2*72))=144 is passed unconditionally to --max-num-seqs and --max-cudagraph-capture-size for every rank.
  5. Compare to the B300 sibling at the equivalent DEP4 tier: it would compute MAX_NUM_SEQS=2*72/4=36, i.e. correctly scaled to its DP world size.
  6. Net effect on B200: every rank is configured with an 8x oversized scheduler batch cap and cudagraph capture ceiling relative to its actual per-rank load.

  Impact: --max-num-seqs and --max-cudagraph-capture-size are upper bounds, not exact allocations — the scheduler and cudagraph bucketing only grow to whatever batch size actually shows up, and B200 (unlike B300, which now explicitly lists every integer capture size) only sets a single --max-cudagraph-capture-size ceiling, so vLLM's default bucket schedule still governs which sizes actually get captured. That means the practical effect is likely limited to a modestly wider capture-size ceiling and some extra scheduler headroom rather than 8x graph memory or a crash. It does not produce incorrect model outputs. However, it is a real fidelity gap relative to the DEP recipe this PR is trying to reproduce, and it makes the B200 DEP numbers not directly comparable to the correctly-scaled B300 DEP tier from the same PR.

  Fix: mirror the B300 branch — add an if [ "$DP_ATTENTION" = "true" ]; then MAX_NUM_SEQS=$((2 * CONC / TP)); else MAX_NUM_SEQS=$((2 * CONC)); fi, plus the same 2*CONC % TP == 0 startup guard B300 uses.

• 🔴 benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh:226-233 — The B200 agentic vLLM recipe (dsv4_fp4_b200_vllm.sh) dropped --tool-call-parser deepseek_v4 and --enable-auto-tool-choice from VLLM_CMD during the array rewrite, even though both flags are still present in the sibling dsv4_fp4_b300_vllm.sh (added in this same PR) and in every other agentic vLLM/SGLang recipe in this directory. Without these flags, vLLM will not enable auto tool-choice or parse DeepSeek-V4's tool-call output into structured tool_calls, which breaks (or silently invalidates) the AgentX tool-use trace-replay workload on B200 while B300 continues to work correctly.

  Extended reasoning...

  What broke: The diff for benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh rewrites the inline vllm serve ... \ invocation into a VLLM_CMD=(...) array. In that rewrite, two flags that existed pre-PR — --tool-call-parser deepseek_v4 and --enable-auto-tool-choice — are deleted (visible as - --tool-call-parser deepseek_v4 / - --enable-auto-tool-choice in the diff hunk) and never reappear anywhere else in the file. The post-PR VLLM_CMD for B200 keeps --tokenizer-mode deepseek_v4 and --reasoning-parser deepseek_v4, but the tool-call-specific flags are simply gone.

  Why this is asymmetric/accidental, not intentional: The sibling script dsv4_fp4_b300_vllm.sh, touched in this exact same PR, targets the same model (DeepSeek-V4-Pro), the same new nightly vLLM image, and the same FLASHINFER_MLA_SPARSE_DSV4 attention backend / deep_gemm_amxf4_mega_moe MoE backend — yet its rewritten VLLM_CMD retains both --tool-call-parser deepseek_v4 and --enable-auto-tool-choice (see the final VLLM_CMD block in dsv4_fp4_b300_vllm.sh). If dropping these flags were a deliberate consequence of the new image or attention backend, B300 would have dropped them too — it didn't. A repo-wide sweep of benchmarks/single_node/agentic/*.sh also shows every other agentic vLLM/SGLang recipe (dsv4_fp8_h200.sh, dsv4_fp4_mi355x_vllm.sh, kimik2.5_*, minimaxm3_*, etc.) passes --tool-call-parser, with tool-using ones also passing --enable-auto-tool-choice. B200 is the sole outlier.

  Code path that triggers it: These agentic recipes run AgentX's trace-replay workload via aiperf against /v1/chat/completions, replaying multi-turn tool-use sessions (tools + tool_choice). vLLM only performs server-side tool-call parsing and validates tool_choice="auto"/"required" requests when started with --enable-auto-tool-choice --tool-call-parser <parser>. There is no fallback for this in benchmark_lib.sh — tool-call handling is configured exclusively through these vllm serve CLI flags.

  Impact / step-by-step proof:

  1. dsv4_fp4_b200_vllm.sh launches vllm serve ... with the VLLM_CMD array shown in the diff — no --enable-auto-tool-choice, no --tool-call-parser.
  2. AgentX's trace replay sends chat-completions requests carrying a tools array and tool_choice: auto (that's the whole point of the agentic-coding benchmark).
  3. Without --enable-auto-tool-choice, vLLM either rejects those requests outright ("auto tool choice requires --enable-auto-tool-choice and --tool-call-parser") or, if it doesn't hard-reject, returns the model's tool invocation as raw unparsed text rather than a structured tool_calls field.
  4. Either outcome breaks the trace replay: outright request failures can trip aiperf's --failed-request-threshold and fail the sweep, or (if requests aren't rejected) the benchmark silently measures a non-functional/non-comparable agentic loop, since downstream turns in the trace depend on structured tool-call results.
  5. Meanwhile dsv4_fp4_b300_vllm.sh, edited in the same PR, keeps both flags, so B300 continues to correctly parse and act on tool calls — producing a divergence where one half of this PR's two new recipes is broken/invalid and the other is fine.

  Fix: Restore --tokenizer-mode deepseek_v4, --tool-call-parser deepseek_v4, --enable-auto-tool-choice, and --reasoning-parser deepseek_v4 together in the B200 VLLM_CMD, matching the B300 sibling's ordering/flags (dsv4_fp4_b300_vllm.sh lines ~225-227).

[claude]

🔴 The new B200 vllm-simple KV-offload config (line 118) hardcodes lazy_offload:false, which per this repo's own note in kimik2.5_int4_h200.sh:47-49 is known to hit a popleft_n AssertionError at low/mid CONC specifically for DSv4 + SimpleCPUOffloadConnector — exactly the combo and CONC range (8-72) swept here. It also uses key cpu_bytes_to_use_per_rank instead of the connector's documented cpu_bytes_to_use key used everywhere else (kimik2.5 and the B300 sibling in this same PR), which likely means the intended CPU_BYTES_PER_RANK budget is silently ignored. Both affect every B200 vllm-simple tier in nvidia-master.yaml.

Extended reasoning...

Issue 1 — lazy_offload:false (high confidence, blocking):

dsv4_fp4_b200_vllm.sh:118 sets kv_connector_extra_config.lazy_offload=false (eager mode) for SimpleCPUOffloadConnector on DeepSeek-V4. The repo's own documented knowledge in kimik2.5_int4_h200.sh:47-49 states plainly: "JSON form (rather than --kv_offloading_backend native shortcut) so we can pass lazy_offload=true. Eager mode hits a popleft_n AssertionError at low/mid CONC on DSv4 + SimpleCPUOffloadConnector." That note is model-specific (DSv4) and connector-specific (SimpleCPUOffloadConnector) — exactly the combination this PR newly wires up for the first time on B200.

nvidia-master.yaml sweeps this backend at conc-list: [8, 12, 16] (pure-TP8 tier) and conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72] (DEP8 tier) — squarely the low/mid CONC range where the assertion was documented. Corroborating that this is an oversight rather than a deliberate change: the B300 sibling script added in this same PR wires up the identical connector but omits lazy_offload entirely (taking the connector's default), so B200's explicit false is inconsistent with both its own sibling and the pre-existing, proven-safe kimik2.5 pattern. There's no evidence in the PR that the new nightly image (vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-515d6e9) fixes this path — the kimik2.5 script, unaffected by this PR, still passes lazy_offload=true to work around it. If the assertion still fires, every B200 vllm-simple server launch crashes at startup/low-CONC.

Issue 2 — cpu_bytes_to_use_per_rank key (lower confidence, flagged for author confirmation):

The same line uses key cpu_bytes_to_use_per_rank, whereas every other use of this connector in the repo — kimik2.5_int4_h200.sh:51 and the B300 sibling added in this same PR (dsv4_fp4_b300_vllm.sh:104) — uses cpu_bytes_to_use. A verifier refuted this as a bug, arguing the two keys could be deliberately different because B200's pure-TP8 vllm-simple tier is a single engine with world_size=TP=8, and per the documented connector behavior (kimik2.5_int4_h200.sh:42-45: "SimpleCPUOffloadConnector internally divides cpu_bytes_to_use by world_size"), passing the already-divided CPU_BYTES_PER_RANK under the auto-dividing cpu_bytes_to_use key would yield TOTAL/64 instead of TOTAL/8 — a real problem the author may have been trying to avoid by using a different, non-dividing key name.

That concern about double-division is legitimate for the pure-TP8 tier, but it doesn't establish that cpu_bytes_to_use_per_rank is an actual recognized connector option — no reference to it exists anywhere else in this repo, and the connector's documented interface (per kimik2.5's own comment) only knows about cpu_bytes_to_use. The correct fix for the pure-TP8 case, following the established kimik2.5 pattern, would have been to pass the un-divided aggregate TOTAL_CPU_DRAM_GB under cpu_bytes_to_use (letting the connector's internal division by world_size=8 produce the right per-rank share), not to invent a new key name. Separately, this same code path is also used for the B200 DEP8 tier, where DP_ATTENTION=true makes each engine's attention world_size=1 (since --tensor-parallel-size 1 --data-parallel-size 8) — there, passing the pre-divided value under the real cpu_bytes_to_use key (matching B300's DEP4 pattern) would be correct with no double-division risk at all. Since kv_connector_extra_config is a permissive dict typically consumed via .get(key, default), the most likely outcome of the unrecognized key is that the connector silently falls back to its default CPU budget rather than the intended TOTAL_CPU_DRAM_GB-derived budget, for both B200 vllm-simple tiers.

Given the refutation raises a plausible alternative explanation that can't be fully ruled out without inspecting the pinned nightly vLLM build's actual connector source, I'm including this as a secondary, lower-confidence item for the author to confirm/fix alongside the lazy_offload issue — which is the well-documented, high-confidence part of this finding and is sufficient on its own to block the B200 vllm-simple tiers.

Proof sketch for lazy_offload: (1) DSv4 B200 vllm-simple tier starts with KV_OFFLOAD_BACKEND=vllm-simple, CONC=8 (lowest swept value). (2) Server launches with --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector",...,"lazy_offload":false,...}'. (3) Per the repo's documented DSv4 + SimpleCPUOffloadConnector behavior, eager mode hits a popleft_n AssertionError at low/mid CONC. (4) The server process crashes during warmup/serving at low CONC, failing that sweep tier — exactly the failure mode kimik2.5 explicitly engineers around by setting lazy_offload=true.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29351479047
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29351479047

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29361309642
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29361309642