Skip to content

Run newly added tests in PRs on TPU7X runners#4422

Draft
darisoy wants to merge 17 commits into
mainfrom
darisoy-tpu7x-pr-new-tests
Draft

Run newly added tests in PRs on TPU7X runners#4422
darisoy wants to merge 17 commits into
mainfrom
darisoy-tpu7x-pr-new-tests

Conversation

@darisoy

@darisoy darisoy commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR enables running newly added or modified tests on TPU v7x runners (tpu7x-unit, tpu7x-integration, tpu7x-post-training-unit) during Pull Request checks (github.event_name == 'pull_request'), in addition to nightly schedules and manual workflow triggers.

Specifically, this PR:

  • Updates .github/workflows/ci_pipeline.yml (tpu7x-tests) to execute on Pull Request events when package build dependencies succeed.
  • Passes flavor through .github/workflows/run_tests_coordinator.yml into .github/workflows/run_tests_against_package.yml.
  • Dynamically appends and newly_added to FINAL_PYTEST_MARKER for non-scheduled TPU7X test runs (INPUTS_FLAVOR =~ ^tpu7x-).
  • Traps PyTest exit code 5 (No tests collected) so pull requests that do not add or modify tests complete cleanly (SUCCESS) in ~45 seconds without failing the CI pipeline.

BUGS: b/530220873
FIXES: b/530220873

Tests

Verified end-to-end on physical TPU v7x runners (tpu7x-8) across 3 scenarios:

  1. Manual Trigger (workflow_dispatch) with 0 Added Tests:
    • Run URL: https://github.com/AI-Hypercomputer/maxtext/actions/runs/29053738050
    • Expected & Verified Behavior: PyTest scans all ~2,600 tests against marker ... and newly_added, matches 0 items (0 selected / 2610 deselected), and returns exit code 5. Our exit code 5 trap catches this, logs "No tests collected matching marker '... and newly_added'. Passing cleanly.", and marks all TPU7X jobs green (SUCCESS) in ~48 seconds.
  2. Pull Request Trigger (pull_request) with 0 Added Tests:
    • Run URL: https://github.com/AI-Hypercomputer/maxtext/actions/runs/29109765191
    • Expected & Verified Behavior: Confirmed that native Pull Request events (PR #4422) automatically trigger all three TPU7X jobs (tpu7x-unit, tpu7x-integration, tpu7x-post-training-unit). Because 0 test functions were touched, PyTest fast-paths (0 selected, exit code 5) and completes green (SUCCESS) in ~48 seconds without reserving long TPU runtimes.
  3. Selective Execution (pull_request) with 1 Added Test:
    • Run URL: https://github.com/AI-Hypercomputer/maxtext/actions/runs/29113377305
    • Expected & Verified Behavior: Confirmed on a temporary test branch adding a newly defined test (test_dummy_tpu7x_newly_added). AST diff detection (get_changed_tests()) isolated the new function and PyTest executed strictly that 1 test on tpu7x-8 hardware (1 selected / 2611 deselected), completing SUCCESS across tpu7x-unit (job 86431286035), tpu7x-integration (job 86431280394), and tpu7x-post-training-unit (job 86431280253).

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@darisoy darisoy changed the title ci: Run newly added tests in PRs on TPU7X runners (b/530220873) Run newly added tests in PRs on TPU7X runners Jul 10, 2026
@darisoy darisoy changed the title Run newly added tests in PRs on TPU7X runners Run newly added tests in PRs on TPU7X runners Jul 10, 2026
@darisoy darisoy closed this Jul 13, 2026
@darisoy darisoy reopened this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant