Audit hierarchical resource-load metrics and define Scheduler-level aggregation granularity

[zhy1658858023]

Background

Issue #107 added the first Proxy -> Scheduler pool_resource reporting path. Proxy can now aggregate Instance pool state into a compact pool-level resource snapshot and report it to Scheduler through register / heartbeat payloads.

Before Scheduler starts using those metrics for load balancing, we need to audit the full hierarchical metric chain and make sure each field is backed by real lower-layer measurements or clearly marked as estimated / unavailable.

The important point is that Scheduler should not consume the same fine-grained metric view as Proxy. Proxy is close to Instances and can use detailed per-Instance queue/resource state. Scheduler should consume coarser, pool-level summaries that are stable enough for cross-pool decisions.

Goal

Audit and refine the resource-load metric hierarchy across:

Instance / Resource Agent / Instance runtime
  -> Proxy InstancePool and local queues
    -> Proxy pool_resource aggregation
      -> Scheduler ProxyPool storage and debug APIs

The output of this issue should answer:

1. Which reported metrics are real-time and supported by bottom-layer modules?
2. Which metrics are currently placeholders, estimates, or not wired to a real data source?
3. Which metrics should remain Proxy-local and which should be reported upward to Scheduler?
4. What coarse-grained Scheduler-level pool resource schema should be used before implementing actual Scheduler load balancing?

This issue is primarily an audit + metric contract cleanup. It should not change Scheduler routing behavior yet.

Metric hierarchy principle

Proxy-level granularity

Proxy can keep fine-grained metrics because it owns local Instance selection, knowledge preparation, ready queues, and direct Instance liveness.

Proxy-level metrics may include:

• per-Instance is_alive
• per-Instance resource freshness
• per-Instance CPU / memory / GPU / GPU memory / network values
• per-Instance admission_state
• per-Instance inflight / qps / gpu_util if actually measured
• local prepare queue depth
• local ready queue depth
• per-queue waiting time or queue age if available
• task-level timing, predictor state, IWS-specific local decision data

These are useful for Proxy UI, local debugging, and future Proxy-side Instance selection.

Scheduler-level granularity

Scheduler should receive only coarser Proxy-pool summaries. Scheduler should not depend on raw per-Instance resource payloads or fine-grained local queue internals.

Scheduler-level metrics should be compact, stable, and comparable across Proxy pools:

{
  "schema_version": 1,
  "proxy_id": "hp_127.0.0.1:8001",
  "generated_at": 1234567890.123,
  "metric_source": {
    "inflight": "proxy_queue|instance_pool|unknown",
    "qps_1m": "proxy_metrics|estimated|unknown",
    "resource": "instance_resource_reports|unknown",
    "queue_depth": "proxy_queue_manager|unknown"
  },
  "instances": {
    "total": 4,
    "alive": 3,
    "stale": 1,
    "with_resource": 3,
    "missing_resource": 0
  },
  "load": {
    "inflight_total": 8,
    "qps_1m_total": 3.4,
    "load_ratio": 0.5,
    "capacity": 16,
    "queue_pressure": 0.25
  },
  "utilization": {
    "cpu_avg": 42.5,
    "cpu_max": 78.0,
    "memory_used_ratio": 0.46,
    "gpu_util_avg": 63.0,
    "gpu_util_max": 95.0,
    "gpu_mem_used_ratio": 0.37,
    "network_rx_mbps_total": 120.0,
    "network_tx_mbps_total": 80.0
  },
  "health": {
    "resource_freshness_s_avg": 1.1,
    "resource_freshness_s_max": 5.4,
    "pool_admission_state": "accepting",
    "data_quality": "complete|partial|missing"
  }
}

The exact schema can evolve, but Scheduler should receive summaries, not raw per-Instance snapshots.

Audit tasks

1. Trace each currently reported load field

Inspect current code paths and document the source of each field in pool_resource:

• instances.total
• instances.alive
• instances.stale
• instances.with_resource
• instances.missing_resource
• load.inflight_total
• load.qps_1m_total
• load.load_ratio
• load.capacity
• load.prepare_queue_depth
• load.ready_queue_depth
• utilization.cpu_avg/max
• utilization.memory_*
• utilization.gpu_util_*
• utilization.gpu_mem_*
• utilization.network_*
• pool_admission_state
• resource_freshness_s

For each field, classify it as:

real-time measured
runtime-maintained counter
derived from measured data
static configured value
placeholder / null
unknown / not wired

2. Verify bottom-layer support for load metrics

Check whether lower layers actually support the metrics we are uploading.

Important examples:

inflight

Questions to answer:

• Is per-Instance inflight updated by actual request execution?
• Is Proxy-level inflight maintained by QueueManager or HeartbeatReporter?
• Does current InstanceLoad.inflight come from a real heartbeat or stay at default 0?
• Should Proxy compute pool-level inflight from its own task queues instead of Instance heartbeat fields?

qps_1m

Questions to answer:

• Is QPS computed anywhere from actual request timestamps?
• Is there a rolling 1-minute counter in Proxy or Instance?
• If not, should this field stay null instead of 0.0 to avoid misleading Scheduler?

gpu_util

Questions to answer:

• Is GPU utilization coming from Resource Agent snapshots?
• Is InstanceLoad.gpu_util still separate from InstanceResource.gpu_util_avg?
• Should Scheduler only consume the resource-derived aggregate and ignore stale placeholder load fields?

Queue depths

Questions to answer:

• Can QueueManager expose prepare queue depth and ready queue depth safely?
• Are these global per Proxy or per Instance slot-worker?
• What is the correct Scheduler-level coarse metric: raw queue depth, queue pressure, or admission state?

3. Define metric source metadata

Add or propose a metric_source / metric_quality section in pool_resource so downstream users know whether a field is real or placeholder.

Example:

{
  "metric_source": {
    "inflight_total": "proxy_queue_manager",
    "qps_1m_total": "unavailable",
    "gpu_util_avg": "instance_resource_snapshot",
    "prepare_queue_depth": "unavailable"
  },
  "metric_quality": {
    "resource": "complete",
    "load": "partial",
    "queue": "missing"
  }
}

This avoids Scheduler accidentally treating missing metrics as zero load.

4. Decide null vs zero semantics

For Scheduler-facing data, use clear semantics:

• 0 means measured real zero.
• null means unavailable / not wired / unknown.
• Missing fields should be tolerated for backward compatibility.

Audit current fields that default to zero and decide whether they should become null when not actually measured.

Candidates:

• qps_1m_total
• inflight_total
• prepare_queue_depth
• ready_queue_depth
• gpu_util_avg
• network_rx_mbps_total
• network_tx_mbps_total

Do not break existing APIs. Prefer optional fields and backward-compatible additions.

5. Separate Proxy-local detail from Scheduler-level summary

Review the current pool_resource shape and decide whether any fields are too fine-grained for Scheduler.

Possible approach:

• Keep detailed per-Instance and raw resource snapshots only in Proxy debug APIs.
• Keep Scheduler-facing pool_resource compact.
• Add a separate local-only endpoint if needed:

GET /debug/pool_resource?detail=local

while Scheduler receives only the coarse summary.

6. Add lightweight debug visibility

Add debug APIs or extend existing ones to make metric provenance visible.

Useful endpoints:

Proxy:
GET /debug/pool_resource
GET /debug/pool_resource_sources

Scheduler:
GET /debug/proxy_pool_resources

/debug/pool_resource_sources can be a simple diagnostic endpoint that reports where each field comes from and whether it is currently wired.

7. Do not implement Scheduler routing yet

This issue must not change:

• Scheduler routing strategy
• Scheduler KDN / Proxy selection behavior
• Proxy Instance selection behavior
• Proxy injection strategy / IWS logic
• KDN behavior
• KVCache injection behavior
• Instance request forwarding behavior

The purpose is to make the metrics trustworthy before using them for scheduling.

Suggested implementation direction

A practical path:

1. Audit current metric sources in code and add a short Markdown or README section documenting the findings.
2. Update build_pool_resource_snapshot() so unavailable fields are represented as null, not fake zero, when appropriate.
3. Add metric_source and/or metric_quality to pool_resource.
4. If QueueManager already exposes safe counters, wire coarse queue pressure into pool_resource; otherwise keep queue fields null and mark source as unavailable.
5. Ensure Scheduler stores and exposes these additions without interpreting them.
6. Add validation steps and small smoke checks.

Validation plan

Compile check

PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile \
  proxy/resource/instance_pool.py \
  proxy/resource/p_control_plane.py \
  proxy/proxy.py \
  proxy/queue/*.py \
  scheduler/resource/control_plane.py \
  scheduler/resource/proxy_pool.py

Runtime check

Start Scheduler, Proxy, and Instance as usual, then inspect:

curl -sS http://127.0.0.1:8002/debug/pool_resource | python3 -m json.tool
curl -sS http://127.0.0.1:7002/debug/proxy_pool_resources | python3 -m json.tool

Expected:

• metric_source or equivalent provenance metadata is present.
• Unavailable fields are null or clearly marked unavailable, not silently reported as measured zero.
• Resource-derived metrics are populated only when Instance resource reports exist.
• Scheduler stores and exposes the pool summary but does not use it for routing.
• Existing demos still run.

Load sanity check

Send a small number of requests through Scheduler and observe whether any load fields move:

curl -sS http://127.0.0.1:7001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-70b",
    "messages": [{"role": "user", "content": "What is CacheRoute?"}],
    "stream": true,
    "RAG": true,
    "Injection_type": "text",
    "max_tokens": 64
  }'

Then check whether inflight_total, qps_1m_total, queue pressure, and freshness fields reflect reality or are marked unavailable.

Acceptance criteria

• There is a clear documented mapping from each Scheduler-facing pool metric to its lower-layer source.
• Metrics that are not actually measured are not reported as fake zeros.
• pool_resource includes source / quality metadata or equivalent diagnostics.
• Scheduler-facing resource granularity is coarser than Proxy-local resource granularity.
• Scheduler continues to only store and expose metrics; no routing behavior changes are introduced.
• Existing Proxy and Scheduler debug APIs remain backward compatible.
• README or docs briefly explain the metric hierarchy and null-vs-zero semantics.