ci(sdk-regression): full 33-SDK matrix via RUN_REGRESSION comment / dispatch with per-SDK ref overrides (PER-9772)

[pranavz28]

What (PER-9772)

Make the SDK regression a usable pre-merge gate for percy/cli changes:

• Triggers — the regression now runs only via:
  • a RUN_REGRESSION comment on a PR (write/admin authors; tests that PR's CLI branch), or
  • workflow_dispatch (defaults to CLI master).
  • The earlier run-sdk-regression label trigger from this PR's first iteration was dropped in review of the final design.
• Full matrix — expanded from 19 to 33 SDKs: added the injection-capable SDKs that were missing from the fan-out (detox, playwright-python/java/dotnet, appium-dotnet, styleguidist) and the App Percy + remaining SDKs (appium-python/java/wd/ruby, maestro-web/app, react-native-app, tosca-dotnet, uipath, xcui-swift). percy-robotframework (archived) and percy-nightmare (downstream hangs) are excluded.
• Per-SDK ref overrides on dispatch — new sdk_refs input, comma-separated repo@branch. Example: branch=master, sdk_refs=percy-cypress@my-fix runs the whole matrix against cli master with percy-cypress's workflow taken from my-fix; unlisted SDKs keep their matrix default ref. Overridden refs are regex-validated before flowing into the downstream dispatch. (Comment path always uses matrix defaults — issue_comment carries no inputs.)
• Dispatch plumbing fixes — job-id lookup paginated to cover the >30-job matrix (and made non-gating); percy-tosca-dotnet/percy-uipath dispatch ci.yml (they have no test.yml); percy-storybook → test-storybook-v10.yml; percy-react-native-app → storybook-rn-ci.yml.
• Removed the App Percy + POA Buildkite job.

Rollout fixes that made the matrix green

Getting to green surfaced 9 repos where percy-bot (the WORKFLOW_DISPATCH_ACTIONS_TOKEN identity) lacked access (now granted), plus 7 SDK-side bugs, all fixed and merged:
percy-selenium-ruby#39 + percy-appium-js#589 (CLI-from-git setup used npx percy without yarn global bin on PATH → pulled the unrelated public percy package), percy-ember#1171 (canvas test asserted attribute order), percy-protractor#734 (local _iframe_shim fallbacks drifted from the canonical sdk-utils contract of #2319), percy-selenium-python#247 (pre-existing coverage debt), percy-maestro-app#17 (no workflow_dispatch inputs declared → 422), percy-styleguidist#26 (ESM test loader mislabeled yarn-linked CJS dists).

Verification

• Full dispatch run with the 33-SDK matrix: 28647243686 — all SDKs green (sole red row was percy-selenium-python before its fix PR ⬆️ Bump karma-rollup-preprocessor from 7.0.6 to 7.0.7 #247 merged).
• sdk_refs parsing unit-tested (empty input → defaults; per-repo exact match, no prefix cross-match; last-wins duplicates; rel/1.2.x-style refs allowed; shell metacharacters rejected).
• Non-default-ref dispatch mechanism verified end-to-end: percy-selenium-ruby test.yml dispatched at a branch ref with the CLI branch input (run 28662964285).

🤖 Generated with Claude Code

[pranavz28]

RUN_regression

[pranavz28]

RUN_REGRESSION

[pranavz28]

🤖 Claude Code Review — stack:pr-review

Summary: Makes the SDK regression workflow a usable pre-merge gate — adds workflow_dispatch (with branch + per-SDK sdk_refs overrides) alongside the existing RUN_REGRESSION comment trigger, expands the matrix from 19→33 SDKs, sets fail-fast: false, paginates the job-id lookup, and routes non-test.yml SDKs to their real workflow files. Single file: .github/workflows/sdk-regression.yml (+119/-15).

Review Table

Priority	Category	Check	Status	Notes
High	Security	No hardcoded secrets	✅ Pass	Only secrets.WORKFLOW_DISPATCH_ACTIONS_TOKEN references; no literals
High	Security	Auth/authz checks present	✅ Pass	Comment path gated to write/admin via check-access; dispatch gated by GitHub's own write requirement
High	Security	Input validation & sanitization	✅ Pass	head_ref regex-validated before use; new sdk_refs parsed from env (not shell-interpolated) and the resolved ref is regex-validated before flowing downstream
High	Security	No IDOR	N/A	—
High	Security	No SQL injection	N/A	—
High	Correctness	Logic correct, edge cases	✅ Pass	Nested workflow_file_name ternary correct; sdk_refs parser handles whitespace-trim, last-wins, exact repo@ match (no prefix cross-match)
High	Correctness	Explicit error handling	✅ Pass	Invalid refs exit 1; job-url intentionally continue-on-error with rationale
High	Correctness	No race conditions	N/A	fail-fast: false is the correct choice here
Medium	Testing	New code has tests	✅ Pass	sdk_refs parsing unit-tested + end-to-end dispatch verified (run 28647243686)
Medium	Testing	Error paths tested	✅ Pass	Empty input, no prefix cross-match, last-wins, dotted refs, shell-metachar rejection
Medium	Testing	Existing tests pass	✅ Pass	Workflow-only change
Medium	Performance	No unbounded fetching	✅ Pass	per_page: 100 fix covers the 35-job matrix (was silently truncating at 30)
Medium	Performance	Background jobs	N/A	—
Medium	Quality	Follows existing patterns	✅ Pass	Consistent with the repo's env-passing + regex-gate hardening (PER-8611)
Medium	Quality	Focused changes	✅ Pass	Single concern, single file
Low	Quality	Meaningful names, no dead code	✅ Pass	—
Low	Quality	Comments explain why	⚠️ Fail	One stale comment (see below)
Low	Quality	No unnecessary deps	✅ Pass	All actions SHA-pinned

Findings

1. Stale comment references a removed "label path" (Low — quality)

• File: .github/workflows/sdk-regression.yml:89-90
• Issue: The comment says "Permission check applies only to the comment path; the label path is already gated by GitHub (only write+ collaborators can label a PR)." But this PR deliberately removed the label trigger — there is no labeled event in on: anymore. The comment describes a code path that no longer exists.
• Suggestion: # Permission check applies only to the comment path; workflow_dispatch is already gated by GitHub (requires repo write access).

2. branch dispatch input can't contain dots, but sdk_refs can (Low — minor inconsistency, not a bug)

• File: .github/workflows/sdk-regression.yml:131 vs :193
• Issue: head_ref (which the dispatch branch input flows into) is validated with ^[a-zA-Z0-9_/\-]+$ (no .), while sdk_refs allows . via ^[a-zA-Z0-9_/.\-]+$. So a dispatch with branch=rel/1.2.x is rejected even though dotted refs are advertised as supported for sdk_refs. Harmless for real PR head branches, but the two validators diverging is a small trap.
• Suggestion: If dotted CLI branches should be dispatchable, align the head_ref regex to also allow .. Otherwise leave it — just be aware of the asymmetry.

Pre-existing (not introduced by this PR): line 150 echo ${{ steps.jobs.outputs.html_url }} references step id jobs, but the id is job-url (line 138) — so it echoes empty. Dates to #1237; purely cosmetic debug echo. Out of scope; could be swept up as a one-char fix.

Security deep-dive

The classic dangerous issue_comment/workflow_dispatch combo is handled correctly:

• The workflow always executes from the default branch (master), so a malicious PR can't alter this file's logic.
• Attacker-controllable head_ref (fork branch name) is passed via env var, regex-validated, and gated by "Break on invalid branch name" before use in the downstream trigger and JSON client_payload — anchored ^...$ regex means no shell/JSON breakout.
• New SDK_REFS is env-passed (never spliced into the shell source), parsed safely, and the resolved ref is regex-validated before reaching the trigger-workflow-and-wait ref: input. read stops at the first newline, so GITHUB_OUTPUT newline-injection isn't reachable.
• Both trigger paths are internal-only (write/admin).

Overall: Pass

Verdict: Approve — security-conscious, well-tested CI change verified green end-to-end. Only nit worth doing before merge is fixing the stale "label path" comment (finding #1); finding #2 is optional.