[WIP] Add CI job priority system

[adibarra]

Priority-aware CI scheduling

GitHub Actions does not natively prioritize self-hosted jobs. This PR adds an opt-in controller that decides which queued GPU job each compatible idle runner may accept.

Key mechanism: GitHub uses AND matching for runs-on labels. A runner must have every label requested by a job.

How dispatch works

With priority scheduling enabled, a job requests labels similar to:

• self-hosted
• b200
• ci-priority-6.000
• ci-queue-a81f23...

Runners permanently carry only their normal hardware and identity labels. They do not permanently carry priority or queue labels.

The controller:

1. Reads queued jobs and online idle runners.
2. Parses each job's numeric priority from its requested labels.
3. Adds queue aging and sorts higher scores first.
4. Finds a compatible runner using existing hardware labels.
5. Temporarily adds the selected job's priority and queue labels.
6. Lets GitHub dispatch the now-exact label match.
7. Removes stale scheduling labels after the runner becomes idle.

The priority label transports the score through GitHub's queued-job API. The queue token uniquely identifies one job. Because every label must match, the token prevents GitHub from selecting another job with the same hardware and priority.

Running jobs are never preempted or relabeled.

Single-node and multinode awareness

Existing runner labels remain compatibility constraints:

• Single-node example: b200
• Multinode example: b200-multinode

A single-node runner cannot accept a multinode job unless it already satisfies the multinode label. The controller schedules the coordinating GitHub runner; existing Slurm launchers continue provisioning worker nodes.

Automatic scoring

All ordinary priorities are numeric, nonnegative, automatic, and uncapped. Higher scores run first.

For PR sweeps, Claude Fable inspects the diff and returns only verified priority facts, such as topology, workload type, precision, speculative decoding, framework, model family, checklist completion, and patchwork.

Fable never receives weights and never calculates a score. ci_priority.py:

1. Rejects unknown facts.
2. Intersects the facts with each matrix job's actual configuration.
3. Reads weights from configs/ci-priority.yaml.
4. Performs deterministic arithmetic.
5. Emits the numeric ci-priority-NUMBER label.

This intersection keeps mixed matrices correct: a PR may contain both vLLM and SGLang jobs, but each job receives only its matching criteria. Main-branch pushes retain deterministic matrix-field scoring without a Fable call.

Malformed or unavailable Fable output conservatively classifies the PR as patchwork. Arbitrary ci-priority:pN overrides are not supported.

Ties are resolved by queue timestamp and job ID. Runner selection preserves scarce exact-match runners when a less-constrained runner can accept the higher-priority job.

Queue tokens

Generated sweep tokens are deterministic hashes of:

• Workflow run namespace
• Matrix path/index
• Canonical job configuration

Manual matrix workflows use the run ID and matrix-cell ID. Tokens are routing identifiers, not credentials or secrets.

Without a queue token, assigning ci-priority-6.000 could release any compatible priority-6.000 job. The unique token makes the controller's assignment target exactly one queued job.

Protected skip_queue

skip_queue is the only absolute bypass mechanism. The workflow keeps the job's ordinary numeric priority and adds a PR-scoped skip request label.

The trusted controller then:

1. Reads the PR timeline.
2. Finds the active label's latest labeling actor.
3. Verifies active SemiAnalysisAI/Core membership.
4. Treats the job's effective priority as absolute only when authorized.

Unauthorized or unverifiable requests retain their numeric priority. REPO_PAT remains inside the trusted controller and is never passed to PR-controlled workflow code.

Required REPO_PAT permissions:

• Actions: Read
• Administration: Write
• Issues: Read
• Organization Members: Read

Safety and failure behavior

• Busy and offline runners are never relabeled.
• Malformed or duplicate scheduler labels fail loudly.
• Transient GitHub API failures retry in watch mode.
• Partial label updates converge on the next controller cycle.
• Controller downtime leaves priority-enabled jobs queued.
• Run exactly one active controller instance.
• Existing routing remains unchanged while disabled.

Activation requires deploying the controller and setting:

PRIORITY_SCHEDULER_ENABLED=true

Verification

• 28 focused tests covering scoring, rounding, aging, authorization, malformed labels, compatibility, exact-runner preservation, and dispatch.
• Changed workflows pass actionlint.
• Simulations verified automatic numeric scoring and unique queue tokens.
• Core-authorized skip_queue outranked p100.000 while retaining its numeric job label.

[claude]

This PR is WIP and touches CI scheduling/authorization logic, so it warrants human review regardless of the inline nits already flagged. Beyond those, I also checked whether the free-text priority/runner workflow_dispatch inputs spliced into the runs-on JSON string (e.g. speedbench-al.yml) could be used to inject extra array elements and redirect a job onto an unintended runner — ruled out: any injected elements become additional required labels under GitHub's AND-matching for self-hosted runners, so they can only narrow matching, never bypass a label or redirect scheduling.

Extended reasoning...

Checked the label-injection candidate (unescaped format() splicing into fromJSON() for runs-on) independently: GitHub self-hosted runner matching requires a runner to possess ALL listed labels (AND semantics), so any extra elements an attacker splices into the JSON array via inputs.priority/inputs.runner can only add more required labels, narrowing which runners match rather than loosening or redirecting to a different/more-privileged runner. Combined with workflow_dispatch already requiring write access to trigger, this isn't an exploitable escalation path. This is in addition to, not a replacement for, the inline nit findings already posted (rounding-mode mismatch, skip-queue YAML config drift, tz-naive plan crash, unvalidated free-text priority input) — none of which are blocking, but the PR overall (skip_queue bypass authorization, priority scheduler affecting real GPU CI dispatch) is exactly the kind of security/infra-adjacent change that should get a human pass before merge.