Skip to content

[AMD][MI355X] DSv4 SGLang agentic: v0.5.15 image, router, HiCache + MTP #2199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seungrokj wants to merge 16 commits into from
+162 −106
Closed

[AMD][MI355X] DSv4 SGLang agentic: v0.5.15 image, router, HiCache + MTP #2199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
64ff5eb
[AMD][MI355X] Update DSv4 SGLang agentic launcher to v0.5.14 image wi…
seungrokj Jul 10, 2026
2c12e4e
[AMD][MI355X] Add dram-utilization 0.80 to DSv4 SGLang agentic sweep …
seungrokj Jul 10, 2026
ffd3717
Update perf-changelog with PR link for DSv4 SGLang agentic entry
seungrokj Jul 10, 2026
f20a7ae
[AMD][MI355X] Remove --context-length and comment out extra sweep con…
seungrokj Jul 10, 2026
b640168
[AMD][MI355X] DSv4 SGLang agentic: use compressed attention, 2x max-r…
seungrokj Jul 10, 2026
9d3806e
[AMD][MI355X] Fix cudaHostRegister OOM: lower hicache-ratio to 4, adj…
seungrokj Jul 10, 2026
0447a00
[AMD][MI355X] DSv4 SGLang agentic: focus sweep on DPA configs, scale …
seungrokj Jul 10, 2026
91a0ea4
[AMD][MI355X] DSv4 SGLang agentic: add non-DPA baseline config (tp=8,…
seungrokj Jul 11, 2026
936eb64
[AMD][MI355X] exclude mia1-p01-g12 from agentic salloc
seungrokj Jul 11, 2026
e59729f
[AMD][MI355X] DSv4 SGLang: narrow DPA sweep ranges, add disagg agenti…
seungrokj Jul 11, 2026
c35020b
[AMD][MI355X] DSv4 SGLang: refactor launcher, rename config key, bump…
seungrokj Jul 14, 2026
7728a9e
[AMD][MI355X] DSv4 SGLang: fix kv-offload-backend to map syntax, add …
seungrokj Jul 14, 2026
637f57e
[AMD][MI355X] DSv4 SGLang: remove disagg agentic hicache recipe
seungrokj Jul 14, 2026
e151b65
[AMD][MI355X] DSv4 SGLang: add vllm-router and mooncake backend to ag…
seungrokj Jul 14, 2026
3e403a5
[AMD][MI355X] DSv4 SGLang: bump agentic-hicache image to v0.5.15
seungrokj Jul 14, 2026
305b965
[AMD][MI355X] DSv4 SGLang: update perf-changelog for image bump and d…
seungrokj Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
237 changes: 142 additions & 95 deletions benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
set -x

# Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on MI355X using SGLang.
# Adapted from benchmarks/single_node/dsv4_fp4_mi355x_sglang.sh (fixed-seq-len
# sibling) with the agentic harness (build_replay_cmd / write_agentic_result_json
# / analyze_benchmark_distributions) swapped in for run_benchmark_serving.
#
# This launcher only supports on-device KV cache.
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=hicache.
#
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
#
# KV_OFFLOADING=dram requires one of these.
# KV_OFFLOAD_BACKEND=hicache.

source "$(dirname "$0")/../../benchmark_lib.sh"

Expand All @@ -25,9 +25,6 @@
export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"
fi

# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# Either way, MODEL_PATH is what the server is launched with.
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
Expand All @@ -43,115 +40,165 @@
resolve_trace_source
install_agentic_deps

require_agentic_kv_offload_none

# Transformers in the container doesn't recognize the `deepseek_v4` model_type.
# PR #23608's fallback in hf_transformers_utils.get_config tries to handle this
# by writing a patched config to /tmp, but in practice isn't catching the error
# in this image. Patch the cached config.json directly instead: set model_type
# to `deepseek_v3` so AutoConfig.from_pretrained succeeds, and keep
# architectures=['DeepseekV4ForCausalLM'] so SGLang dispatches to its native
# DSv4 model class (python/sglang/srt/models/deepseek_v4.py).
python3 << PYEOF
import json
from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="$MODEL", filename="config.json")
with open(path) as f:
config = json.load(f)
if config.get("model_type") == "deepseek_v4":
config["model_type"] = "deepseek_v3"
with open(path, "w") as f:
json.dump(config, f, indent=2)
print(f"Patched {path}: model_type deepseek_v4 -> deepseek_v3")
else:
print(f"No patch needed: model_type is {config.get('model_type')!r}")
PYEOF

# DSv4 FP4-experts path. Mirrors the env block in the fixed-seq-len sibling
# (benchmarks/single_node/dsv4_fp4_mi355x_sglang.sh), which tracks the active
# block in python/run_dsv4.sh on the amd/deepseek_v4 branch:
# SGLANG_DSV4_FP4_EXPERTS=True -> route experts through FP4 kernels
# SGLANG_FORCE_TRITON_MOE_FP8=0 -> dispatch MoE through aiter and apply
# the swiglu_limit clamp in the triton
# MoE fallback path.
export SGLANG_REASONING_EFFORT=max
export SGLANG_OPT_USE_FUSED_COMPRESS=true
export SGLANG_OPT_USE_OLD_COMPRESSOR=true
export SGLANG_OPT_USE_TILELANG_SWA_PREPARE=false
export SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=false
export SGLANG_OPT_USE_FUSED_HASH_TOPK=false
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=false
export SGLANG_OPT_USE_TILELANG_MHC_PRE=false
export SGLANG_OPT_USE_TILELANG_MHC_POST=false
export SGLANG_OPT_USE_AITER_MHC_PRE=true
export SGLANG_OPT_USE_AITER_MHC_POST=true
export SGLANG_ENABLE_THINKING=1
export SGLANG_USE_AITER=1
export SGLANG_USE_ROCM700A=1
export SGLANG_TOPK_TRANSFORM_512_TORCH=0
export SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1
export SGLANG_DSV4_FP4_EXPERTS=True
export SGLANG_OPT_DPSK_V4_RADIX=0
export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=false
export SGLANG_OPT_USE_FUSED_STORE_CACHE=false
export SGLANG_FORCE_TRITON_MOE_FP8=0
export SGLANG_HACK_FLASHMLA_BACKEND=tilelang
export SGLANG_OPT_USE_TILELANG_INDEXER=true
export SGLANG_OPT_USE_TRITON_SWA_PREPARE=true

# ---- Server config ----------------------------------------------------------
SERVER_LOG="$RESULT_DIR/server.log"
mkdir -p "$RESULT_DIR"

# Parallelism: pure TP, TP+EP, or DEP (DP-attn + EP). Matches the dsv4 b200
# vllm agentic launcher so the agentic sweep can probe both interactivity and
# throughput regimes.
PARALLEL_ARGS=(--tensor-parallel-size "$TP")
export SGLANG_ENABLE_UNIFIED_RADIX_TREE=1
export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1

CACHE_ARGS=()
if agentic_kv_offload_enabled; then
# HiCache config — https://lmsysorg.mintlify.app/cookbook/autoregressive/DeepSeek/DeepSeek-V4
case "${KV_OFFLOAD_BACKEND:-}" in
hicache)
HICACHE_RATIO=4
HICACHE_WRITE_POLICY="write_through"
HICACHE_IO_BACKEND="direct"
HICACHE_MEM_LAYOUT="page_first_direct"
CACHE_ARGS=(
--enable-hierarchical-cache
--hicache-ratio "$HICACHE_RATIO"
--hicache-write-policy "$HICACHE_WRITE_POLICY"
--hicache-io-backend "$HICACHE_IO_BACKEND"
--hicache-mem-layout "$HICACHE_MEM_LAYOUT"
)
echo "HiCache DSv4 CPU tier: ratio=$HICACHE_RATIO, write_policy=$HICACHE_WRITE_POLICY, io_backend=$HICACHE_IO_BACKEND, mem_layout=$HICACHE_MEM_LAYOUT"
;;
*)
echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: hicache)" >&2
exit 1
;;
esac
fi

# ---- LLM server config ----------------------------------------------------------
USE_SGLANG_ROUTER=false
SGLANG_BACKEND_PORT="$PORT"
ROUTER_LOG="$RESULT_DIR/router.log"
MEM_FRACTION_STATIC=0.95
CHUNKED_PREFILL_SIZE=16384
if [ "$DP_ATTENTION" = "true" ]; then
USE_SGLANG_ROUTER=true
export AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=true
SGLANG_BACKEND_PORT=$((PORT + 1))
SGLANG_ROUTER_METRICS_PORT=$((PORT + 10000))
SGLANG_ROUTER_CMD=(python3 -m sglang_router.launch_router)

export SGLANG_SHARED_EXPERT_TP1=1
export SGLANG_DP_SHARED_EXPERT_LOCAL=1
export SGLANG_DP_USE_GATHERV=1
export SGLANG_DP_USE_REDUCE_SCATTER=1
export GPU_MAX_HW_QUEUES=5

CHUNKED_PREFILL_SIZE=$((8192 * TP))
PARALLEL_ARGS+=(
--dp "$TP"
--enable-dp-attention
--enable-prefill-delayer
--enable-two-batch-overlap
)
fi

if [ "${EP_SIZE:-1}" -gt 1 ]; then
PARALLEL_ARGS+=(--ep-size "$EP_SIZE")
fi

# --max-running-requests is per-engine. With DP-attn each DP engine handles
# only CONC/$TP sequences in steady state (the agentic harness load-balances
# users across DP ranks), so size the per-engine cap to that.
# Pure TP is a single engine and sees all CONC sequences itself.
if [ "$DP_ATTENTION" = "true" ]; then
PER_ENGINE_MAX_RUNNING=$(( CONC / TP ))
[ "$PER_ENGINE_MAX_RUNNING" -lt 1 ] && PER_ENGINE_MAX_RUNNING=1
else
PER_ENGINE_MAX_RUNNING=$CONC
fi
# AgentX concurrency counts live session trees, not individual requests.
# Allow subagent fan-out to exceed CONC without clipping request bursts.
MAX_RUNNING_REQUESTS=$((2 * CONC))
CUDA_GRAPH_MAX_BS=$CONC
[ "$CUDA_GRAPH_MAX_BS" -gt 128 ] && CUDA_GRAPH_MAX_BS=128

echo "Starting sglang server..."
python3 -m sglang.launch_server \
--model-path "$MODEL_PATH" --served-model-name "$MODEL" \
--host=0.0.0.0 \
--port "$PORT" \
"${PARALLEL_ARGS[@]}" \
--trust-remote-code \
--attention-backend compressed \
--max-running-requests "$PER_ENGINE_MAX_RUNNING" \
--cuda-graph-max-bs "$PER_ENGINE_MAX_RUNNING" \
--page-size 256 \
--chunked-prefill-size 8192 \
--disable-shared-experts-fusion \
--tool-call-parser deepseekv4 \
--reasoning-parser deepseek-v4 \
--chat-template "$(dirname "$0")/../chat_templates/deepseek_v4_thinking.jinja" \
--watchdog-timeout 1800 > "$SERVER_LOG" 2>&1 &
export SGLANG_DEFAULT_THINKING=1
export SGLANG_DSV4_REASONING_EFFORT=max
export SGLANG_USE_ROCM700A=0
export SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton
export AITER_BF16_FP8_MOE_BOUND=0

PARALLEL_ARGS=(--tensor-parallel-size "$TP")
METRICS_ARGS=(--enable-metrics)
SPEC_ARGS=()

Check failure on line 122 in benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh

View check run for this annotation

Claude / Claude Code Review

PARALLEL_ARGS reassignment wipes DP-attention/EP-size flags 

PARALLEL_ARGS is appended to at lines 95/104 (--dp/--enable-dp-attention/--enable-prefill-delayer/--enable-two-batch-overlap for DP_ATTENTION=true, and --ep-size for EP_SIZE>1), but line 119 does a plain reassignment `PARALLEL_ARGS=(--tensor-parallel-size "$TP")` that wipes out everything appended earlier. As a result, every dp-attn:true / ep-size>1 row in the new sweep (all the agentic-hicache DPA/HiCache rows) launches sglang as plain TP while the router is still started with --dp-aware/consis
Comment on lines +108 to +122

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 PARALLEL_ARGS is appended to at lines 95/104 (--dp/--enable-dp-attention/--enable-prefill-delayer/--enable-two-batch-overlap for DP_ATTENTION=true, and --ep-size for EP_SIZE>1), but line 119 does a plain reassignment PARALLEL_ARGS=(--tensor-parallel-size "$TP") that wipes out everything appended earlier. As a result, every dp-attn:true / ep-size>1 row in the new sweep (all the agentic-hicache DPA/HiCache rows) launches sglang as plain TP while the router is still started with --dp-aware/consistent_hashing, expecting DP ranks that no longer exist.

Extended reasoning...

The bug: In benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh, PARALLEL_ARGS is built up incrementally with += in two places:

  • Inside the if [ "$DP_ATTENTION" = "true" ] block: PARALLEL_ARGS+=(--dp "$TP" --enable-dp-attention --enable-prefill-delayer --enable-two-batch-overlap)
  • Inside the if [ "${EP_SIZE:-1}" -gt 1 ] block: PARALLEL_ARGS+=(--ep-size "$EP_SIZE")

Both of these run before the array is initialized. Then, unconditionally, a later line does:

PARALLEL_ARGS=(--tensor-parallel-size "$TP")

This is a plain assignment, not +=. It discards every element that was appended above and replaces the array with just --tensor-parallel-size $TP. Since this line executes after both append blocks, PARALLEL_ARGS never contains --dp, --enable-dp-attention, --enable-prefill-delayer, --enable-two-batch-overlap, or --ep-size by the time it's expanded into SGLANG_CMD ("${PARALLEL_ARGS[@]}") and handed to sglang.launch_server.

Why nothing catches this: the script runs under set -euo pipefail, but appending to an unset array with += does not error under nounset — bash implicitly creates the array on first append — so the DP/EP blocks execute silently without any indication that their output is about to be thrown away.

Why this matters for this PR specifically: the diff that moved the base PARALLEL_ARGS=(--tensor-parallel-size "$TP") initialization is new — in the pre-PR version of this file, that same line existed before the DP-attention append block, so it worked correctly. The refactor relocated the PARALLEL_ARGS=(...) init to right before the SGLANG_CMD construction, after the DP/EP blocks, turning a correct pattern into a silent clobber.

Concrete walkthrough for one of the new sweep rows, e.g. { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 48, 64] }:

  1. DP_ATTENTION=true → PARALLEL_ARGS+=(--dp 8 --enable-dp-attention --enable-prefill-delayer --enable-two-batch-overlap). PARALLEL_ARGS is now (--dp 8 --enable-dp-attention --enable-prefill-delayer --enable-two-batch-overlap).
  2. EP_SIZE is unset/1, so the EP block is skipped.
  3. USE_SGLANG_ROUTER=true is set, SGLANG_BACKEND_PORT=$((PORT+1)), the router command is prepared.
  4. Later: PARALLEL_ARGS=(--tensor-parallel-size "$TP") executes — PARALLEL_ARGS is now just (--tensor-parallel-size 8), and everything from step 1 is gone.
  5. SGLANG_CMD is built with "${PARALLEL_ARGS[@]}" → the server launches as plain TP8, no DP-attention, no two-batch-overlap.
  6. But USE_SGLANG_ROUTER=true still causes the router to start with --dp-aware and --policy consistent_hashing, routing to a backend that has zero DP ranks — a topology mismatch between what the router expects and what the server actually is.

Impact: every dp-attn:true row in the new dsv4-fp4-mi355x-sglang-agentic-hicache sweep (conc 16/32/48/64 for plain DPA, and 80/96 for the HiCache+DRAM-offload rows) silently degrades to plain TP8, defeating the entire point of the DPA/HiCache sweep added by this PR, while the router/backend pairing is left inconsistent.

Fix: either change the base-init line to PARALLEL_ARGS+=(--tensor-parallel-size "$TP"), or move the base initialization above the DP-attention/EP-size append blocks (restoring the original ordering).

SGLANG_CMD=(
python3 -m sglang.launch_server
--model-path "$MODEL_PATH"
--served-model-name "$MODEL"
--host 0.0.0.0
--port "$SGLANG_BACKEND_PORT"
--trust-remote-code
"${PARALLEL_ARGS[@]}"
--attention-backend compressed
--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS"
--max-running-requests "$MAX_RUNNING_REQUESTS"
--mem-fraction-static "$MEM_FRACTION_STATIC"
--swa-full-tokens-ratio 0.10
--page-size 256
--kv-cache-dtype fp8_e4m3
--chunked-prefill-size "$CHUNKED_PREFILL_SIZE"
--disable-shared-experts-fusion
--tool-call-parser deepseekv4
--reasoning-parser deepseek-v4
--chat-template "$(dirname "$0")/../chat_templates/deepseek_v4_thinking.jinja"
--watchdog-timeout 1800
"${METRICS_ARGS[@]}"
"${SPEC_ARGS[@]}"
"${CACHE_ARGS[@]}"
)

printf '%q ' "${SGLANG_CMD[@]}" | tee "$RESULT_DIR/sglang_command.txt"
printf '\n' | tee -a "$RESULT_DIR/sglang_command.txt"

{
echo "=== SGLANG_* env vars at launch ==="
env | grep -E '^SGLANG_' | sort
echo "==================================="
} | tee "$SERVER_LOG"

echo "Starting SGLang server for MI355X..."
"${SGLANG_CMD[@]}" >> "$SERVER_LOG" 2>&1 &
SERVER_PID=$!
echo "Server PID: $SERVER_PID"

wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"
capture_cache_metrics() {
{
echo "=== SGLang cache metrics snapshot $(date --iso-8601=seconds) ==="
curl -fsS "http://localhost:$SGLANG_BACKEND_PORT/metrics" 2>/dev/null \
| grep -E '^(sglang:(cache_hit_rate|cached_tokens_total|prompt_tokens_total|hicache_host_used_tokens|hicache_host_total_tokens|token_usage|num_requests_running|num_requests_waiting))' \
|| true
echo "============================================================"
} >> "$SERVER_LOG"
}

wait_for_server_ready --port "$SGLANG_BACKEND_PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

if [ "$USE_SGLANG_ROUTER" = "true" ]; then
echo "Starting SGLang router on port $PORT for $TP DP ranks..."
"${SGLANG_ROUTER_CMD[@]}" \
--worker-urls "http://localhost:$SGLANG_BACKEND_PORT" \
--policy consistent_hashing \
--request-id-headers x-correlation-id \
--dp-aware \
--host 0.0.0.0 \
--port "$PORT" \
--prometheus-host 127.0.0.1 \
--prometheus-port "$SGLANG_ROUTER_METRICS_PORT" \
--connect-timeout-secs 900 \
--request-timeout-secs 14400 \
--disable-health-check \
--disable-retries > "$ROUTER_LOG" 2>&1 &
ROUTER_PID=$!
echo "Router PID: $ROUTER_PID"
wait_for_server_ready --port "$PORT" --server-log "$ROUTER_LOG" --server-pid "$ROUTER_PID"
fi

if [ "${#METRICS_ARGS[@]}" -gt 0 ]; then
capture_cache_metrics
trap capture_cache_metrics EXIT
fi

# ---- Run benchmark ----------------------------------------------------------
build_replay_cmd "$RESULT_DIR"
REPLAY_CMD+=" --server-metrics http://localhost:$SGLANG_BACKEND_PORT/metrics"

run_agentic_replay_and_write_outputs "$RESULT_DIR"
17 changes: 7 additions & 10 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2341,13 +2341,8 @@ dsr1-fp4-mi355x-sglang-disagg-mtp:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=1"

# DSv4-Pro FP4 on MI355X via SGLang. Uses a rocm720 mi35x image built off the
# amd/deepseek_v4 branch in sgl-project/sglang; the SHA is encoded in the
# image tag, so bumping sglang is just an image tag bump here. Sweeps
# DP-attention on/off and EP=8.

dsv4-fp4-mi355x-sglang-agentic:
image: rocm/sgl-dev:rocm720-mi35x-0363e6c-20260509-DSv4
dsv4-fp4-mi355x-sglang-agentic-hicache:
image: lmsysorg/sglang-rocm:v0.5.15-rocm700-mi35x-20260713
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:mi355x-amds
Expand All @@ -2356,9 +2351,11 @@ dsv4-fp4-mi355x-sglang-agentic:
multinode: false
scenarios:
agentic-coding:
- search-space:
- { tp: 8, kv-offloading: none, conc-list: [16, 32, 64] }
- { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [64, 128, 256] }
- dram-utilization: 0.80
search-space:
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8] }
- { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 48, 64] }
- { tp: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [80, 96] }

# MiniMax-M3 MXFP8 MI355X recipe:
# https://github.com/vllm-project/recipes/commit/2a3728ed9892debfd767a72a58ebc90b33f186e5
Expand Down
12 changes: 12 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4750,3 +4750,15 @@
- "Image: lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928"
- "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137

- config-keys:
- dsv4-fp4-mi355x-sglang-agentic-hicache
description:
- "Bump image to lmsysorg/sglang-rocm:v0.5.15-rocm700-mi35x-20260713"
- "Align launcher env vars and server args with fixed-seq-len sibling (dsv4 attention backend, fp8_e4m3 kv-cache, disable-radix-cache, cuda-graph-max-bs, DP-attention exports, two-batch-overlap)"
- "Add SGLang router for DP-attention configs (consistent-hashing, dp-aware, correlation-id routing)"
- "Add HiCache KV offloading support"
- "Add SGLANG_CMD array pattern with command logging and env-var dump"
- "Add capture_cache_metrics for pre/post-benchmark cache stats"
- "Sweep conc=48 across TP8 +/- DPA +/- HiCache"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2146

Check warning on line 4764 in perf-changelog.yaml

View check run for this annotation

Claude / Claude Code Review

perf-changelog.yaml entry links to wrong PR number 

The new perf-changelog.yaml entry for `dsv4-fp4-mi355x-sglang-agentic-hicache` sets `pr-link` to pull/2146, but this changelog entry is actually introduced by this PR, #2199. Every other entry in the file links to the PR that introduced it, so this breaks the convention; please update the link to #2199 (and it looks like the description bullets were also copy-pasted from #2146 and don't match this diff, e.g. "disable-radix-cache" vs. the actual `SGLANG_ENABLE_UNIFIED_RADIX_TREE=1` change, and "S
Comment on lines +4753 to +4764

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new perf-changelog.yaml entry for dsv4-fp4-mi355x-sglang-agentic-hicache sets pr-link to pull/2146, but this changelog entry is actually introduced by this PR, #2199. Every other entry in the file links to the PR that introduced it, so this breaks the convention; please update the link to #2199 (and it looks like the description bullets were also copy-pasted from #2146 and don't match this diff, e.g. "disable-radix-cache" vs. the actual SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 change, and "Sweep conc=48" vs. the actual conc-lists).

Extended reasoning...

What the bug is

The new perf-changelog.yaml entry for config-key dsv4-fp4-mi355x-sglang-agentic-hicache (added at the bottom of the file, lines 4753-4764) sets:

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2146

But per the PR metadata, this changelog entry is being introduced by this PR, #2199. Every other entry in perf-changelog.yaml links to the PR that actually introduced it (the immediately preceding entries link to #2144, #2169, #2137, etc.), so the convention throughout the file is consistent, and pull/2146 breaks it.

Why this happened (copy-paste leftover)

This looks like the whole changelog block — link included — was copy-pasted from an earlier, unrelated PR (#2146) rather than authored fresh for #2199. The description bullets corroborate this: they describe things that do not match this PR's actual diff:

  • The entry claims "disable-radix-cache", but the launcher (benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh) never passes --disable-radix-cache; it instead exports SGLANG_ENABLE_UNIFIED_RADIX_TREE=1, which enables a unified radix tree — the opposite framing.
  • The entry claims "Sweep conc=48 across TP8 +/- DPA +/- HiCache", but the actual configs/amd-master.yaml search-space for this key sweeps conc-list: [1, 2, 4, 8], [16, 32, 48, 64], and [80, 96] — conc=48 only appears inside the DPA-no-HiCache row, not as a dedicated cross-cutting sweep, and the HiCache row itself only covers [80, 96].

So the whole entry (link + several bullets) appears lifted wholesale from the earlier #2146 hicache work rather than describing this PR's actual changes.

Impact

This is a documentation/traceability field only — pr-link is not consumed by any benchmark execution path, so nothing breaks functionally, no runs fail, and no results are miscomputed. The impact is limited to anyone using the changelog to trace which PR introduced a given recipe change (e.g. for auditing or bisecting a perf regression) getting sent to the wrong PR (#2146 instead of #2199).

Step-by-step proof

  1. Open perf-changelog.yaml lines 4753-4764 — the new entry for dsv4-fp4-mi355x-sglang-agentic-hicache.
  2. Note pr-link: .../pull/2146.
  3. Check the PR metadata for the current PR: it is [AMD][MI355X] DSv4 SGLang agentic: v0.5.15 image, router, HiCache + MTP  #2199, and the diff for this PR is exactly what adds this changelog block (git blame/git log -p on perf-changelog.yaml would show this hunk was introduced in [AMD][MI355X] DSv4 SGLang agentic: v0.5.15 image, router, HiCache + MTP  #2199, not [AMD][MI355X] DSv4 SGLang agentic: v0.5.15 image, router, HiCache #2146).
  4. Compare adjacent entries in the same file (e.g. the entry just above, pr-link ...Add Qwen3.5 FP8 GB300 disaggregated multinode SGLang benchmarks via Dynamo / 新增 Qwen3.5 FP8 GB300 分离式多节点 SGLang（Dynamo）基准测试 #2137) — each of those links to the PR whose diff actually introduced that entry, confirming the file-wide convention.
  5. Cross-check the entry's own description against this PR's diff: "disable-radix-cache" is absent from the launcher script's SGLANG_CMD array, and there is no standalone "conc=48" sweep in the config — both signs that the description (and by extension the pr-link) were copied from a different PR's changelog entry rather than written for [AMD][MI355X] DSv4 SGLang agentic: v0.5.15 image, router, HiCache + MTP  #2199.

How to fix

Change pr-link to https://github.com/SemiAnalysisAI/InferenceX/pull/2199, and while touching it, reconcile the description bullets to reflect what this PR actually changed (unified radix tree env var, fp8_e4m3 KV cache, SGLang router for DP-attention, SGLANG_CMD array pattern, capture_cache_metrics, and the real conc-list sweep), dropping the stale "disable-radix-cache" / "Sweep conc=48" bullets.

2 changes: 1 addition & 1 deletion runners/launch_mi355x-amds.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ else
# mia1-p01-g09: pyxis broken (persistently fails to create container filesystem)
# mia1-p01-g11: docker.sock permissions denied (cluster-cleanup step fails)
# Both have been root-caused via #1431/#1432/#1440/#1441/#1443 sweep failures.
salloc --partition=$PARTITION --exclude=mia1-p01-g09,mia1-p01-g11 --gres=gpu:$GPU_COUNT --exclusive --cpus-per-task=128 --time=500 --no-shell --job-name="$RUNNER_NAME"
salloc --partition=$PARTITION --exclude=mia1-p01-g09,mia1-p01-g11,mia1-p01-g12 --gres=gpu:$GPU_COUNT --exclusive --cpus-per-task=128 --time=500 --no-shell --job-name="$RUNNER_NAME"
JOB_ID=$(squeue --name="$RUNNER_NAME" -h -o %A | head -n1)

srun --jobid=$JOB_ID bash -c "docker stop \$(docker ps -a -q)"
Expand Down
Loading