Skip to content

test(signature): serialize registry-auth env tests to end cross-test env bleed (JEF-412)#234

Merged
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-412-flaky-test-docker_io_variants_match_the_index_v1_config_key
Jul 12, 2026
Merged

test(signature): serialize registry-auth env tests to end cross-test env bleed (JEF-412)#234
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-412-flaky-test-docker_io_variants_match_the_index_v1_config_key

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Closes JEF-412

Problem

policies::signature::auth::tests::docker_io_variants_match_the_index_v1_config_key (and its whole env-touching sibling family) passed in isolation but flaked under full-suite cargo test parallelism (reproduced ~40% failure rate over 25 runs).

Shared global + siblings that bled it: RegistryAuth::from_env reads three process-global env vars — PROTECTOR_REGISTRY_USERNAME, PROTECTOR_REGISTRY_PASSWORD, and PROTECTOR_REGISTRY_AUTH_FILE. Rust's default test harness runs tests as parallel threads in one process, so a set_var/remove_var in one test was observed by another mid-flight. The bleeding was mutual across resolves_per_image_across_multiple_registries, docker_io_variants_match_the_index_v1_config_key, ghcr_private_image_still_resolves_basic_from_the_auth_file, and especially env_override_applies_across_all_registries (it sets PROTECTOR_REGISTRY_USERNAME/PROTECTOR_REGISTRY_PASSWORD, which any concurrent test then saw as a global override).

CI stayed green only because it runs cargo nextest, which isolates each test in its own process — the stale test-file header even claimed nextest isolation, but cargo test gives none.

Fix (chosen: serialize + restore — option 2)

Test-hermeticity only; no production behavior change. from_env() is the exact API the webhook and the sweep call, and the tests must keep exercising its real env-reading — so rather than refactor the env read out of the production hot path (option 1), every env-mutating test now routes through an RAII EnvGuard:

  • takes a shared static ENV_LOCK: Mutex<()> so only one env-mutating test runs at a time;
  • snapshots the three vars, then clears them to a known-clean state;
  • restores the snapshot on Dropeven on panic — so a failing assertion can never leak creds into a sibling;
  • recovers from lock poisoning so one legitimate assertion failure does not cascade into spurious failures.

The var list lives in one CRED_ENV_VARS array that drives both the clear and the restore. This folds away the old repeated clear_cred_env() + unsafe { set_var } boilerplate at each test site. No new dependency (no serial_test).

Verification

Deterministic under parallel execution — 30/30 green:

for i in (seq 1 30); cargo test -p protector --lib policies::signature::auth::tests -- --test-threads=8; end
# 30/30 ok (was ~40% flaky before the fix)

Full signature policy area also green (109 passed). cargo fmt --check clean, cargo clippy --lib --tests -- -D warnings clean. Test file is 242 lines (well under the 1000-line cap).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP

…env bleed (JEF-412)

RegistryAuth::from_env reads three process-global env vars
(PROTECTOR_REGISTRY_USERNAME / PROTECTOR_REGISTRY_PASSWORD /
PROTECTOR_REGISTRY_AUTH_FILE). Rust's default harness runs tests as parallel
threads in one process, so one case's set_var/remove_var was observed by a
sibling mid-flight — making docker_io_variants_match_the_index_v1_config_key
(and its whole env-touching family) flaky under `cargo test`. CI stays green
only because it runs nextest, which isolates each test in its own process.

Fix (test-hermeticity only; no production change): route every env-mutating
test through an RAII EnvGuard that takes a shared static Mutex, snapshots the
three vars, clears them to a known state, and restores the snapshot on drop —
even on panic — so no test can see another's mutation. The lock recovers from
poisoning so one legitimate assertion failure can't cascade.

Verified deterministic: 30/30 green under
`cargo test -p protector --lib policies::signature::auth::tests -- --test-threads=8`
(was ~40% flaky before).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
@thejefflarson thejefflarson merged commit a544db1 into main Jul 12, 2026
5 checks passed
@thejefflarson thejefflarson deleted the thejefflarson/jef-412-flaky-test-docker_io_variants_match_the_index_v1_config_key branch July 12, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant