feat(py_test): add opt-in safeguard against silently-passing tests

[thirtyseven]

Summary

Implements #3824.

py_test runs the main module directly and does not automatically invoke a test runner (e.g. unittest/pytest). A common pitfall is to define test classes/functions but forget to run them, so the test silently passes without executing anything.

This adds a validation action to py_test that statically analyzes the main module with ast and fails the build if the module's top-level body is "inert" — only definitions, imports, assignments, and docstrings, with nothing that actually runs tests (such as an if __name__ == "__main__": guard invoking a runner).

Because this is technically a breaking change, it is gated behind a new flag, //python/config_settings:validate_test_main, with values auto/enabled/ disabled. The default is auto, which currently resolves to disabled, and can be flipped to enabled in rules_python 3.0.

Notes/limitations:

• Only applies to py_test targets with a main source file; main_module targets are not checked.
• Enabling requires the exec-tools toolchain (default for hermetic toolchains); if it's missing, the build fails with a message pointing at the disable flag.

Test plan

• Analysis tests (tests/base_rules/py_test) verify the PyValidateTestMain
  action and _validation output group are present when enabled and absent
  when disabled.
• Unit test (tests/validate_test_main) covers the ast classification
  (inert vs. runs-something).
• Bazel-in-bazel integration test (tests/integration/validate_test_main)
  builds real py_test targets with the flag and asserts an inert test fails
  the build with a descriptive error when enabled, while a correct test builds;
  and that an inert test builds when disabled / by default.

[gemini-code-assist]

Code Review

This pull request introduces an opt-in safeguard (validate_test_main) to prevent py_test targets from silently passing when they do not actually execute any tests. It implements a static analysis validator using Python's ast module to detect inert main modules and fail the build if no active test runner is invoked. The feedback suggests extending the validator's inert node list to include ast.Global and ast.TypeAlias (for Python 3.12+ compatibility) and adding defensive checks in Starlark to safely access the exec_tools toolchain.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To prevent potential Starlark evaluation errors (such as struct has no field 'exec_tools'), we should defensively check if exec_tools_toolchain actually has the exec_tools attribute and that it is not None before attempting to access exec_tools.exec_interpreter.

    exec_tools_toolchain = ctx.toolchains[EXEC_TOOLS_TOOLCHAIN_TYPE]\n    if (exec_tools_toolchain == None or \n        not hasattr(exec_tools_toolchain, 'exec_tools') or \n        exec_tools_toolchain.exec_tools == None or \n        exec_tools_toolchain.exec_tools.exec_interpreter == None):\n        fail(\n            'Validating py_test main modules requires the exec tools toolchain ' +\n            'with an exec interpreter, but none was found. Either register one ' +\n            'or set --@rules_python//python/config_settings:validate_test_main=disabled.',\n        )

[thirtyseven]

I'll leave this as-is for consistency with the existing precompile.bzl, which accesses exec_tools_toolchain.exec_tools.precompiler with the same pattern (only a == None check on the toolchain). The exec tools toolchain rule always constructs a PyExecToolsInfo, so .exec_tools is guaranteed present when the toolchain resolves; adding a hasattr guard here would diverge from the rest of the codebase. Happy to revisit if there's a known case of a custom exec_tools_toolchain_type that returns a bare ToolchainInfo.

[gemini-code-assist]

Code Review

This pull request introduces an opt-in build-time validation safeguard (validate_test_main) for py_test targets to prevent them from silently passing when they define tests but fail to invoke a test runner. The validation statically analyzes the main module using an AST parser to ensure it contains active statements. The review feedback highlights a critical gap where compound statements like ast.If (e.g., if TYPE_CHECKING:) and ast.Try (e.g., try/except ImportError:) are not recursively checked, allowing inert files to bypass validation. Suggestions were provided to recursively check these blocks and add corresponding unit tests.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[thirtyseven]

/gemini PTAL

[gemini-code-assist]

Code Review

This pull request introduces an opt-in safeguard (validate_test_main) for py_test targets to statically analyze the main module and prevent tests from silently passing without executing any test code. It adds a validator script (py_test_main_validator.py) using Python's ast module to detect inert top-level statements, along with corresponding build settings, flags, integration tests, and unit tests. The review feedback correctly identifies a critical issue where all assignments (e.g., exit_code = unittest.main()) are unconditionally treated as inert, which would cause false positives. The reviewer suggests dynamically checking assignments and expressions for active nodes (like function calls) using ast.walk and adding corresponding unit tests to verify this behavior.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Add unit tests to verify that active assignments (e.g., exit_code = unittest.main()) and active expressions (e.g., sys.exit(unittest.main())) are correctly identified as active and allowed by the validator.

        )

    def test_definitions_with_active_assignment_runs_tests(self):
        self.assertTrue(
            _runs_tests(
                """
                import unittest

                class MyTest(unittest.TestCase):
                    def test_foo(self):
                        pass

                exit_code = unittest.main()
                """
            )
        )

    def test_definitions_with_active_expression_runs_tests(self):
        self.assertTrue(
            _runs_tests(
                """
                import unittest

                class MyTest(unittest.TestCase):
                    def test_foo(self):
                        pass

                sys.exit(unittest.main())
                """
            )
        )

[thirtyseven]

Added in 66811ad — test_definitions_with_active_assignment_runs_tests and test_definitions_with_active_expression_runs_tests.

[thirtyseven]

/gemini PTAL

[gemini-code-assist]

Code Review

This pull request introduces an opt-in safeguard (validate_test_main) to prevent py_test targets from silently passing when they define tests but fail to invoke a test runner. It implements a static analysis tool (py_test_main_validator.py) using Python's ast module to check for active statements in the test's main module and integrates this check as a build-time validation action. Feedback on the changes includes explicitly handling ast.Assert statements as inert in the validator (with corresponding unit tests) and appending to the _validation output group in py_executable.bzl rather than overwriting it to prevent conflicts with other validation actions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[thirtyseven]

/gemini PTAL

[gemini-code-assist]

Code Review

This pull request introduces an opt-in build-time validation safeguard for py_test targets to prevent them from silently passing when no tests are actually executed. It adds the --@rules_python//python/config_settings:validate_test_main flag and a static analysis tool (py_test_main_validator.py) that parses the test's main module using the ast module to ensure it contains active statements (like invoking a test runner) rather than only inert definitions or imports. The feedback points out a potential false positive where a test runner is invoked directly inside an if condition (e.g., if unittest.main().wasSuccessful():), and suggests updating the validator to check if the condition expression itself runs code.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.