diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index c81d31e44..4acf7714f 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -55,6 +55,7 @@ In this major release, several modifications were made to the PTB's workflow tem ## Documentation +* #789: Consolidated the metrics and Sonar documentation to reflect the current PTB reporting flow * #828: Removed the legacy migration page and merged the useful guidance into getting started ## Security diff --git a/doc/user_guide/features/metrics/collecting_metrics.rst b/doc/user_guide/features/metrics/collecting_metrics.rst index 2545d3706..0571eedb6 100644 --- a/doc/user_guide/features/metrics/collecting_metrics.rst +++ b/doc/user_guide/features/metrics/collecting_metrics.rst @@ -1,8 +1,8 @@ -Collecting Metrics -================== +Measuring Code Quality +====================== -The PTB allows you to collect various metrics on the quality of your project -regarding Coverage, Security, and Static Code Analysis. +The PTB uses various Nox sessions incl. linting and tests to collect code-quality data +and upload the results to Sonar for aggregation and review. .. toctree:: :maxdepth: 2 @@ -13,16 +13,29 @@ regarding Coverage, Security, and Static Code Analysis. .. _generated_metrics: -Generating Metrics -++++++++++++++++++ +Overview +++++++++ -For each metric, there is a dedicated nox session, generating one or multiple +The PTB code-quality flow has three stages: + +1. Run linting and test sessions that generate artifact files. +2. Collect and validate those artifacts in ``report.yml``. +3. Upload the consolidated results with ``sonar:check``. + +The individual tools still produce the raw data, but Sonar is the main place +where PTB projects review the combined results in CI. + + +Generate Code Quality Results ++++++++++++++++++++++++++++++ + +For each result type, there is a dedicated nox session, generating one or multiple files and based on a selected external Python tool. +------------------------------------+-----------------------------+--------------+ | Nox session | Generated files | Based on | +====================================+=============================+==============+ -| ``lint:code`` | ``lint.txt``, ``lint.json`` | ``pylint`` | +| ``lint:code`` | ``.lint.json`` | ``pylint`` | +------------------------------------+-----------------------------+--------------+ | ``lint:security`` | ``.security.json`` | ``bandit`` | +------------------------------------+-----------------------------+--------------+ @@ -31,28 +44,21 @@ files and based on a selected external Python tool. | ``test:integration -- --coverage`` | ``.coverage`` | ``coverage`` | +------------------------------------+-----------------------------+--------------+ -These metrics are computed for each element in your build matrix, e.g. for each +These results are computed for each element in your build matrix, e.g. for each Python version defined in the `PROJECT_CONFIG` of the ``noxconfig.py`` file. -The GitHub workflows of your project can: - -* Use a build matrix, e.g. using different Python versions as shown above -* Define multiple test sessions, e.g. for distinguishing fast vs. slow or expensive tests. +The GitHub workflows of your project can use a build matrix and multiple test +workflows, for example to distinguish fast and slow tests. -Reporting Metrics -+++++++++++++++++ +Report Code Quality Results ++++++++++++++++++++++++++++ -The PTB uses only the metrics associated with the Python version specified by +The PTB uses only the results associated with the Python version specified by :meth:`exasol.toolbox.config.BaseConfig.minimum_python_version`. -Nox session ``sonar:check`` uploads the :ref:`findings ` to -:ref:`SonarQube ` for aggregation and additional static -code analysis, presenting the results in Sonar's `feature-rich UI -`__. - -The CI workflow ``report.yml`` runs ``sonar:check`` after two additional Nox -sessions collect the artifacts from various jobs: +In CI, workflow ``report.yml`` downloads the previously generated artifacts and +then runs the following Nox sessions: +--------------------------+----------------------------------------------------------+ | Nox session | Actions | @@ -66,3 +72,22 @@ sessions collect the artifacts from various jobs: | | * Checks that each file contains the expected attributes | | | for that file type | +--------------------------+----------------------------------------------------------+ +| ``sonar:check`` | * Creates ``ci-coverage.xml`` from ``.coverage`` | +| | * Uploads lint, security, and coverage data to Sonar | ++--------------------------+----------------------------------------------------------+ + +After that, Sonar becomes the main review surface for combined PTB code-quality results. + + +Configure Sonar ++++++++++++++++ + +See :ref:`sonarqube_analysis` for the repository, secret, and +``pyproject.toml`` configuration needed for Sonar. + + +Handle Findings ++++++++++++++++ + +See :ref:`ignore_findings` if a Sonar finding cannot be fixed immediately and +must be explicitly accepted or ignored. diff --git a/doc/user_guide/features/metrics/ignore_findings.rst b/doc/user_guide/features/metrics/ignore_findings.rst index 3cd5595d6..6ae59ce9b 100644 --- a/doc/user_guide/features/metrics/ignore_findings.rst +++ b/doc/user_guide/features/metrics/ignore_findings.rst @@ -3,14 +3,14 @@ Ignoring Sonar Findings ======================= -In rare cases, Sonar might report a finding, you cannot fix or on which you -can agree with your reviewers to accept or ignore it. Please only choose this -approach as a last resort. +In rare cases, Sonar might report a finding that you cannot fix immediately or +that reviewers agree to accept. Please only choose this approach as a last +resort. -As Sonar reports only findings generated by other tools there are often -multiple IDs you can use for referring to a particular issue. E.g. security -findings can be reported by ``bandit`` and hence for such a finding there is a -bandit ID as well as a Sonar ID. +Remember that Sonar often displays findings originating from another tool. That +means a single issue can have both a Sonar identifier and a tool-specific +identifier. For example, security findings reported by ``bandit`` can usually +be referred to by their Bandit rule code as well. Example ------- @@ -25,15 +25,15 @@ on "Why is this an issue?", you will find references like `Bandit `__ *website*. -In this case, the bandit error code is ``B603``, which you can ignore via a -comment in the source code or, alternatively, accept in the Sonar UI. +In this case, the Bandit error code is ``B603``. When possible, prefer handling +the finding at the source-tool level so that the reason is documented in code. Ignoring a Finding Via a Source Code Comment -------------------------------------------- -The recommended way of ignoring such a finding is to append a comment to the -relevant line of code: +If the originating tool supports it, the recommended way of ignoring a finding +is to add an explicit source code comment on the affected line: .. code-block:: python diff --git a/doc/user_guide/features/metrics/sonar.rst b/doc/user_guide/features/metrics/sonar.rst index 4856fda1a..0df4cce75 100644 --- a/doc/user_guide/features/metrics/sonar.rst +++ b/doc/user_guide/features/metrics/sonar.rst @@ -4,21 +4,30 @@ SonarQube Analysis ================== The PTB supports using `SonarQube Cloud `__ -to analyze, visualize, & track linting, security, & coverage. All of our Python projects -should be evaluated against the `Exasol Way`_ and subscribe to the +to analyze, visualize, and track linting, security, and coverage results that +PTB uploads via ``sonar:check``. + +All of our Python projects should be evaluated against the `Exasol Way`_ and +subscribe to the `Clean as You Code `__ -methodology. If code modified in a PR does not satisfy the aforementioned criteria, the -SonarQube analysis fails. +methodology. If code modified in a pull request does not satisfy the configured +criteria, the Sonar analysis fails. -The PTB includes instructions to set up a GitHub bot to display the results of the -Sonar analysis in your pull requests as a stylized comment and workflow result. -Section :ref:`sonar_configuration` gives instructions for public and private repositories. +The PTB workflow ``report.yml`` runs ``sonar:check`` after the linting and test +jobs have produced their artifacts. For the end-to-end PTB code-quality flow, see +:doc:`collecting_metrics`. .. _sonar_configuration: Configuration +++++++++++++ +To use Sonar with PTB, configure: + +* GitHub access and secrets +* the Sonar project +* the project-specific Sonar settings in ``pyproject.toml`` and ``noxconfig.py`` + .. _configure_sonar_public_project: **Public** GitHub Repository @@ -44,7 +53,7 @@ In Sonar In the Code """"""""""" -#. In the ``noxconfig.py``, the relative path to the project's source code is defined with ``Config.sonar_code_path``. +#. In ``noxconfig.py``, the relative path to the project's source code is defined with ``Config.sonar_code_path``. #. Add the following to the project's file ``pyproject.toml`` .. code-block:: toml