From e19860b5b564726f60eb3694e4e157d1f58a0235 Mon Sep 17 00:00:00 2001 From: jana-selva Date: Mon, 22 Jun 2026 15:58:02 +0530 Subject: [PATCH 1/3] Fix issue 789: consolidate metrics and Sonar docs --- .../features/metrics/collecting_metrics.rst | 61 +++++++++++++------ .../features/metrics/ignore_findings.rst | 22 +++---- doc/user_guide/features/metrics/sonar.rst | 25 +++++--- 3 files changed, 71 insertions(+), 37 deletions(-) diff --git a/doc/user_guide/features/metrics/collecting_metrics.rst b/doc/user_guide/features/metrics/collecting_metrics.rst index 2545d3706..de64a282f 100644 --- a/doc/user_guide/features/metrics/collecting_metrics.rst +++ b/doc/user_guide/features/metrics/collecting_metrics.rst @@ -1,8 +1,8 @@ Collecting Metrics ================== -The PTB allows you to collect various metrics on the quality of your project -regarding Coverage, Security, and Static Code Analysis. +The PTB collects code-quality data from its Nox sessions and then uploads the +relevant results to Sonar for aggregation and review. .. toctree:: :maxdepth: 2 @@ -13,8 +13,21 @@ regarding Coverage, Security, and Static Code Analysis. .. _generated_metrics: -Generating Metrics -++++++++++++++++++ +Overview +++++++++ + +The PTB metrics flow has three stages: + +1. Run linting and test sessions that generate artifact files. +2. Collect and validate those artifacts in ``report.yml``. +3. Upload the consolidated results with ``sonar:check``. + +The individual tools still produce the raw data, but Sonar is the main place +where PTB projects review the combined results in CI. + + +Generate Metrics +++++++++++++++++ For each metric, there is a dedicated nox session, generating one or multiple files and based on a selected external Python tool. @@ -22,7 +35,7 @@ files and based on a selected external Python tool. +------------------------------------+-----------------------------+--------------+ | Nox session | Generated files | Based on | +====================================+=============================+==============+ -| ``lint:code`` | ``lint.txt``, ``lint.json`` | ``pylint`` | +| ``lint:code`` | ``.lint.json`` | ``pylint`` | +------------------------------------+-----------------------------+--------------+ | ``lint:security`` | ``.security.json`` | ``bandit`` | +------------------------------------+-----------------------------+--------------+ @@ -34,25 +47,18 @@ files and based on a selected external Python tool. These metrics are computed for each element in your build matrix, e.g. for each Python version defined in the `PROJECT_CONFIG` of the ``noxconfig.py`` file. -The GitHub workflows of your project can: - -* Use a build matrix, e.g. using different Python versions as shown above -* Define multiple test sessions, e.g. for distinguishing fast vs. slow or expensive tests. +The GitHub workflows of your project can use a build matrix and multiple test +workflows, for example to distinguish fast and slow tests. -Reporting Metrics -+++++++++++++++++ +Report Metrics ++++++++++++++++ The PTB uses only the metrics associated with the Python version specified by :meth:`exasol.toolbox.config.BaseConfig.minimum_python_version`. -Nox session ``sonar:check`` uploads the :ref:`findings ` to -:ref:`SonarQube ` for aggregation and additional static -code analysis, presenting the results in Sonar's `feature-rich UI -`__. - -The CI workflow ``report.yml`` runs ``sonar:check`` after two additional Nox -sessions collect the artifacts from various jobs: +In CI, workflow ``report.yml`` downloads the previously generated artifacts and +then runs the following Nox sessions: +--------------------------+----------------------------------------------------------+ | Nox session | Actions | @@ -66,3 +72,22 @@ sessions collect the artifacts from various jobs: | | * Checks that each file contains the expected attributes | | | for that file type | +--------------------------+----------------------------------------------------------+ +| ``sonar:check`` | * Creates ``ci-coverage.xml`` from ``.coverage`` | +| | * Uploads lint, security, and coverage data to Sonar | ++--------------------------+----------------------------------------------------------+ + +After that, Sonar becomes the main review surface for combined PTB metrics. + + +Configure Sonar ++++++++++++++++ + +See :ref:`sonarqube_analysis` for the repository, secret, and +``pyproject.toml`` configuration needed for Sonar. + + +Handle Findings ++++++++++++++++ + +See :ref:`ignore_findings` if a Sonar finding cannot be fixed immediately and +must be explicitly accepted or ignored. diff --git a/doc/user_guide/features/metrics/ignore_findings.rst b/doc/user_guide/features/metrics/ignore_findings.rst index 3cd5595d6..6ae59ce9b 100644 --- a/doc/user_guide/features/metrics/ignore_findings.rst +++ b/doc/user_guide/features/metrics/ignore_findings.rst @@ -3,14 +3,14 @@ Ignoring Sonar Findings ======================= -In rare cases, Sonar might report a finding, you cannot fix or on which you -can agree with your reviewers to accept or ignore it. Please only choose this -approach as a last resort. +In rare cases, Sonar might report a finding that you cannot fix immediately or +that reviewers agree to accept. Please only choose this approach as a last +resort. -As Sonar reports only findings generated by other tools there are often -multiple IDs you can use for referring to a particular issue. E.g. security -findings can be reported by ``bandit`` and hence for such a finding there is a -bandit ID as well as a Sonar ID. +Remember that Sonar often displays findings originating from another tool. That +means a single issue can have both a Sonar identifier and a tool-specific +identifier. For example, security findings reported by ``bandit`` can usually +be referred to by their Bandit rule code as well. Example ------- @@ -25,15 +25,15 @@ on "Why is this an issue?", you will find references like `Bandit `__ *website*. -In this case, the bandit error code is ``B603``, which you can ignore via a -comment in the source code or, alternatively, accept in the Sonar UI. +In this case, the Bandit error code is ``B603``. When possible, prefer handling +the finding at the source-tool level so that the reason is documented in code. Ignoring a Finding Via a Source Code Comment -------------------------------------------- -The recommended way of ignoring such a finding is to append a comment to the -relevant line of code: +If the originating tool supports it, the recommended way of ignoring a finding +is to add an explicit source code comment on the affected line: .. code-block:: python diff --git a/doc/user_guide/features/metrics/sonar.rst b/doc/user_guide/features/metrics/sonar.rst index 4856fda1a..ab076939e 100644 --- a/doc/user_guide/features/metrics/sonar.rst +++ b/doc/user_guide/features/metrics/sonar.rst @@ -4,21 +4,30 @@ SonarQube Analysis ================== The PTB supports using `SonarQube Cloud `__ -to analyze, visualize, & track linting, security, & coverage. All of our Python projects -should be evaluated against the `Exasol Way`_ and subscribe to the +to analyze, visualize, and track linting, security, and coverage results that +PTB uploads via ``sonar:check``. + +All of our Python projects should be evaluated against the `Exasol Way`_ and +subscribe to the `Clean as You Code `__ -methodology. If code modified in a PR does not satisfy the aforementioned criteria, the -SonarQube analysis fails. +methodology. If code modified in a pull request does not satisfy the configured +criteria, the Sonar analysis fails. -The PTB includes instructions to set up a GitHub bot to display the results of the -Sonar analysis in your pull requests as a stylized comment and workflow result. -Section :ref:`sonar_configuration` gives instructions for public and private repositories. +The PTB workflow ``report.yml`` runs ``sonar:check`` after the linting and test +jobs have produced their artifacts. For the end-to-end PTB metrics flow, see +:doc:`collecting_metrics`. .. _sonar_configuration: Configuration +++++++++++++ +To use Sonar with PTB, configure: + +* GitHub access and secrets +* the Sonar project +* the project-specific Sonar settings in ``pyproject.toml`` and ``noxconfig.py`` + .. _configure_sonar_public_project: **Public** GitHub Repository @@ -44,7 +53,7 @@ In Sonar In the Code """"""""""" -#. In the ``noxconfig.py``, the relative path to the project's source code is defined with ``Config.sonar_code_path``. +#. In ``noxconfig.py``, the relative path to the project's source code is defined with ``Config.sonar_code_path``. #. Add the following to the project's file ``pyproject.toml`` .. code-block:: toml From d196a5456599777943f1b2b0167c26e33f8220e2 Mon Sep 17 00:00:00 2001 From: jana-selva Date: Mon, 22 Jun 2026 18:02:50 +0530 Subject: [PATCH 2/3] Add changelog entry for issue 789 --- doc/changes/unreleased.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index dcc4d9128..959dfe9b5 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -22,6 +22,10 @@ so ITDE-related test flows use the configured Exasol baseline and unit-test help * #744: Extracted shared minimum-version selection logic into `minimum_declared_version()` +## Documentation + +* #789: Consolidated the metrics and Sonar documentation to reflect the current PTB reporting flow + ## Security * #867: Fixed zizmor linting results From 6f462cd61d326af324924b822bfb9d77270502e4 Mon Sep 17 00:00:00 2001 From: jana-selva Date: Tue, 23 Jun 2026 11:58:35 +0530 Subject: [PATCH 3/3] Refine code quality wording for issue 789 --- .../features/metrics/collecting_metrics.rst | 26 +++++++++---------- doc/user_guide/features/metrics/sonar.rst | 2 +- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/user_guide/features/metrics/collecting_metrics.rst b/doc/user_guide/features/metrics/collecting_metrics.rst index de64a282f..0571eedb6 100644 --- a/doc/user_guide/features/metrics/collecting_metrics.rst +++ b/doc/user_guide/features/metrics/collecting_metrics.rst @@ -1,8 +1,8 @@ -Collecting Metrics -================== +Measuring Code Quality +====================== -The PTB collects code-quality data from its Nox sessions and then uploads the -relevant results to Sonar for aggregation and review. +The PTB uses various Nox sessions incl. linting and tests to collect code-quality data +and upload the results to Sonar for aggregation and review. .. toctree:: :maxdepth: 2 @@ -16,7 +16,7 @@ relevant results to Sonar for aggregation and review. Overview ++++++++ -The PTB metrics flow has three stages: +The PTB code-quality flow has three stages: 1. Run linting and test sessions that generate artifact files. 2. Collect and validate those artifacts in ``report.yml``. @@ -26,10 +26,10 @@ The individual tools still produce the raw data, but Sonar is the main place where PTB projects review the combined results in CI. -Generate Metrics -++++++++++++++++ +Generate Code Quality Results ++++++++++++++++++++++++++++++ -For each metric, there is a dedicated nox session, generating one or multiple +For each result type, there is a dedicated nox session, generating one or multiple files and based on a selected external Python tool. +------------------------------------+-----------------------------+--------------+ @@ -44,17 +44,17 @@ files and based on a selected external Python tool. | ``test:integration -- --coverage`` | ``.coverage`` | ``coverage`` | +------------------------------------+-----------------------------+--------------+ -These metrics are computed for each element in your build matrix, e.g. for each +These results are computed for each element in your build matrix, e.g. for each Python version defined in the `PROJECT_CONFIG` of the ``noxconfig.py`` file. The GitHub workflows of your project can use a build matrix and multiple test workflows, for example to distinguish fast and slow tests. -Report Metrics -+++++++++++++++ +Report Code Quality Results ++++++++++++++++++++++++++++ -The PTB uses only the metrics associated with the Python version specified by +The PTB uses only the results associated with the Python version specified by :meth:`exasol.toolbox.config.BaseConfig.minimum_python_version`. In CI, workflow ``report.yml`` downloads the previously generated artifacts and @@ -76,7 +76,7 @@ then runs the following Nox sessions: | | * Uploads lint, security, and coverage data to Sonar | +--------------------------+----------------------------------------------------------+ -After that, Sonar becomes the main review surface for combined PTB metrics. +After that, Sonar becomes the main review surface for combined PTB code-quality results. Configure Sonar diff --git a/doc/user_guide/features/metrics/sonar.rst b/doc/user_guide/features/metrics/sonar.rst index ab076939e..0df4cce75 100644 --- a/doc/user_guide/features/metrics/sonar.rst +++ b/doc/user_guide/features/metrics/sonar.rst @@ -14,7 +14,7 @@ methodology. If code modified in a pull request does not satisfy the configured criteria, the Sonar analysis fails. The PTB workflow ``report.yml`` runs ``sonar:check`` after the linting and test -jobs have produced their artifacts. For the end-to-end PTB metrics flow, see +jobs have produced their artifacts. For the end-to-end PTB code-quality flow, see :doc:`collecting_metrics`. .. _sonar_configuration: