fix(metrics) + perf(health-report): correct separator LOO, scope block-impact to changed files by aspala · Pull Request #58 · num42/codeqa-action

aspala · 2026-06-06T13:53:15Z

Why

health-report takes ages on real codebases (a 647-file project ran 30min+ per invocation, and it runs the analysis twice — head + base snapshot). While diagnosing that, a pre-existing correctness bug in the leave-one-out (LOO) path surfaced. This PR fixes both.

1. fix(metrics): remove incorrect `separator_counts` subtractive LOO

separator_counts.analyze_loo/2 (shipped since #30) computed the wrong leave-one-out values for dot_count/underscore_count/etc. in refactoring potentials.

Root cause: the block-impact analyzer reconstructs "file minus block" from normalized structural tokens, which collapse inter-token whitespace (normalize(content) joined != content). analyze_loo subtracted separator counts of that whitespace-collapsed block string from a baseline computed on the original file — mixing two different tokenizations.

Evidence: the subtractive value diverged from a full re-analyze of the reconstructed file in 0/50 real blocks (pricing.ex) and 15/15 in mega_service.ex.

Fix: drop analyze_loo/2 so the analyzer falls back to a full re-analyze (consistent with the other 22 file metrics). Verified the fix yields 50/50 matches.

Adds a regression guard (test/codeqa/metrics/file/subtractive_loo_test.exs) that runs every analyze_loo metric against a full re-analyze on real, deeply-nested sample files, plus a non-vacuous check asserting the fixtures actually collapse whitespace on reconstruction. Proven to go red on the reintroduced bug and green on the fix.

Note: with no metric currently implementing analyze_loo/2, the main guard test's inner loop is a no-op today — its value is forward-looking (it bites the moment someone reintroduces a bad analyze_loo). The non-vacuous fixture check still runs real assertions.

2. perf(health-report): scope per-node block impact to changed files

The block-impact analyzer ran a full per-node LOO (each node: a whole-file re-analyze + two cosine passes over ~1400 behaviors) for every file in the repo, then TopBlocks discarded the nodes of every file not in the diff — almost all the work, thrown away, twice per run.

Fix: scope per-node computation to the PR's changed files. The codebase aggregate, baseline cosines, similarity and near-duplicate metrics are still computed over every file, so the codebase scope, overall grade, and each changed block's codebase-scope delta are unchanged. Only the per-file node output — read for changed files alone — is skipped for the rest.

nil (no base ref / standalone run) keeps the all-files behavior.

Measured on a 647-file project with 18 changed files: head analysis 135s vs an extrapolated ~30min unscoped (~13×). The remaining time is now dominated by near_duplicate_blocks_codebase (43s, runs over all files) — a separate future lever.

Tests assert the scoped run's nodes are bit-identical to the unscoped run for in-scope files, empty for the rest, and that the codebase aggregate is untouched.

Verification

895 tests, 0 failures
Two independent review passes confirmed no consumer reads nodes for non-changed files in a way that reaches rendered output (the two repo-wide readers feed worst_offenders/top_nodes, which no formatter renders and which the report path sets to [])
Edge cases checked: standalone run, empty diff, added/deleted/renamed files, base-snapshot path matching

separator_counts.analyze_loo/2 subtracted separator counts from an original-file baseline using a block string rebuilt from normalized structural tokens, which collapse inter-token whitespace. The baseline came from analyze(original) while the block came from the normalized token stream, so the subtraction mixed two different tokenizations. Empirically the LOO value diverged from a full re-analyze of the reconstructed file in 0/50 real blocks (pricing.ex) and 15/15 in mega_service.ex — every refactoring-potential delta for these counts has been wrong since the metric gained analyze_loo in #30. Drop analyze_loo/2 so the block-impact analyzer falls back to a full re-analyze, which is consistent with the other 22 file metrics. Add a regression guard (subtractive_loo_test) that runs every analyze_loo metric against a full re-analyze on real sample files with deep nesting, plus a non-vacuous check asserting the fixtures actually collapse whitespace on reconstruction. Verified the guard goes red on the reintroduced bug and green on the fix.

The block-impact analyzer ran a full per-node leave-one-out (each node: a whole-file re-analyze plus two cosine passes over ~1400 behaviors) for every file in the repo, then TopBlocks discarded the nodes of every file not in the diff. On a 647-file project this meant ~30min+ of work per run, almost all of it thrown away — and health-report does it twice (head and base snapshot). Scope per-node computation to the PR's changed files. The codebase aggregate, baseline cosines, similarity and near-duplicate metrics are still computed over every file, so the codebase scope, overall grade and each changed block's codebase-scope delta are unchanged. Only the per-file node output — read for changed files alone — is skipped for the rest. Wiring: resolve changed_files up front (before analysis) and pass them as :node_paths through analyze_codebase to BlockImpactAnalyzer, which feeds out-of-scope files "" so compute_nodes_timed short-circuits to no nodes. nil (no base ref / standalone run) keeps the all-files behavior. Measured on a 647-file project with 18 changed files: head analysis 135s vs an extrapolated ~30min unscoped. Tests assert the scoped run's nodes are bit-identical to the unscoped run for in-scope files, empty for the rest, and that the codebase aggregate is untouched.

github-actions · 2026-06-06T13:56:01Z

🟠 Code Health: C+ (65/100)

190 files · codeqa-action · 2026-06-06

Combined metric scores use cosine similarity: +1 = metric profile perfectly matches healthy pattern for this behavior, 0 = no signal, −1 = anti-pattern detected. Mapped to 0–100 using breakpoints (approx: ≥0.5→A, ≥0.2→B, ≥0.0→C, ≥−0.3→D, <−0.3→F); actual letter grades use the full 15-step scale.

%%{init: {'theme': 'neutral'}}%%
xychart-beta
    title "Code Health Scores"
    x-axis ["Readability", "Complexity", "Structure", "Duplication", "Naming", "Magic Numbers", "Combined Metrics"]
    y-axis "Score" 0 --> 100
    bar [95, 31, 87, 48, 96, 100, 67]

Readability       ███████████████████░   95  🟢 A
Complexity        ██████░░░░░░░░░░░░░░   31  🔴 D-
Structure         █████████████████░░░   87  🟢 A-
Duplication       ██████████░░░░░░░░░░   48  🟠 C-
Naming            ███████████████████░   96  🟢 A
Magic Numbers     ████████████████████  100  🟢 A
Combined Metrics  █████████████░░░░░░░   67  🔴 D+

github-actions · 2026-06-06T13:56:02Z

🔍 Top Likely Issues (cosine similarity)

Most negative cosine = file's metric profile best matches this anti-pattern.

Behavior	Cosine	Score
`dependencies.low_coupling`	-0.56	-12.18
`file_structure.single_responsibility`	-0.51	-12.24
`file_structure.line_count_under_300`	-0.44	-9.40
`code_smells.no_dead_code_after_return`	-0.43	-34.50
`scope_and_assignment.used_only_once`	-0.42	-14.93
`scope_and_assignment.shadowed_by_inner_scope`	-0.34	-4.83
`file_structure.line_length_under_120`	-0.31	-8.41
`scope_and_assignment.declared_close_to_use`	-0.26	-1.18
`code_smells.no_debug_print_statements`	-0.25	-2.21
`variable_naming.loop_var_is_single_letter`	-0.23	3.58

🟢 Readability — A (95/100)

Codebase averages: flesch_adapted=97.76, fog_adapted=4.81, avg_tokens_per_line=9.48, avg_line_length=35.65

Metric	Value	Score
readability.flesch_adapted	97.76	100
readability.fog_adapted	4.81	100
readability.avg_tokens_per_line	9.48	73
readability.avg_line_length	35.65	100

🔴 Complexity — D- (31/100)

Codebase averages: difficulty=41.15, effort=237517.76, volume=4071.80, estimated_bugs=1.36

Metric	Value	Score
halstead.difficulty	41.15	42
halstead.effort	237517.76	0
halstead.volume	4071.80	46
halstead.estimated_bugs	1.36	46

🟢 Structure — A- (87/100)

Codebase averages: branching_density=0.14, mean_depth=3.87, avg_function_lines=8.36, max_depth=9.16, max_function_lines=20.27, variance=6.90, avg_param_count=1.15, max_param_count=2.01

Metric	Value	Score
branching.branching_density	0.14	76
indentation.mean_depth	3.87	88
function_metrics.avg_function_lines	8.36	89
indentation.max_depth	9.16	87
function_metrics.max_function_lines	20.27	90
indentation.variance	6.90	100
function_metrics.avg_param_count	1.15	100
function_metrics.max_param_count	2.01	100

🟠 Duplication — C- (48/100)

Codebase averages: redundancy=0.59, bigram_repetition_rate=0.54, trigram_repetition_rate=0.37

Metric	Value	Score
compression.redundancy	0.59	58
ngram.bigram_repetition_rate	0.54	38
ngram.trigram_repetition_rate	0.37	40

🟢 Naming — A (96/100)

Codebase averages: entropy=0.89, mean=6.66, variance=18.94, avg_sub_words_per_id=1.18

Metric	Value	Score
casing_entropy.entropy	0.89	100
identifier_length_variance.mean	6.66	100
identifier_length_variance.variance	18.94	85
readability.avg_sub_words_per_id	1.18	100

🟢 Magic Numbers — A (100/100)

Codebase averages: density=0.00

Metric	Value	Score
magic_number_density.density	0.00	100

🔴 Combined Metrics — D+ (67/100)

Category	Score	Grade
Code Smells	44	🔴 D+
Consistency	80	🟡 B+
Dependencies	19	🔴 E+
Documentation	83	🟡 B+
Error Handling	92	🟢 A-
File Structure	48	🟠 C-
Function Design	81	🟡 B+
Naming Conventions	90	🟢 A-
Scope And Assignment	29	🔴 D-
Testing	83	🟡 B+
Type And Value	90	🟢 A-
Variable Naming	74	🟡 B

🔴 Code Smells — D+ (44/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
no_dead_code_after_return	-0.43	25	D-
no_debug_print_statements	-0.25	33	D-
no_fixme_comments	0.26	74	B

🟡 Consistency — B+ (80/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
consistent_function_style	0.36	80	B+

🔴 Dependencies — E+ (19/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
low_coupling	-0.56	19	E+

🟡 Documentation — B+ (83/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
file_has_module_docstring	0.29	76	B
function_has_docstring	0.44	86	A-
docstring_is_nonempty	0.44	86	A-

🟢 Error Handling — A- (92/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
error_message_is_descriptive	0.52	90	A-
does_not_swallow_errors	0.61	92	A-
returns_typed_error	0.70	94	A

🟠 File Structure — C- (48/100)

Cosine similarity scores for 5 behaviors.

Behavior	Cosine	Score	Grade
single_responsibility	-0.51	21	E+
line_count_under_300	-0.44	24	E+
line_length_under_120	-0.31	30	D-
has_consistent_indentation	0.26	74	B
no_magic_numbers	0.57	91	A-

🟡 Function Design — B+ (81/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
is_less_than_20_lines	0.34	79	B+
no_magic_numbers	0.38	82	B+
has_verb_in_name	0.40	83	B+

Worst offender: lib/codeqa/ast/parsing/signal_stream.ex:1-20 (20 lines)

🟢 Naming Conventions — A- (90/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
function_name_is_not_single_word	0.51	90	A-

🔴 Scope And Assignment — D- (29/100)

Cosine similarity scores for 3 behaviors.

Behavior	Cosine	Score	Grade
used_only_once	-0.42	25	D-
shadowed_by_inner_scope	-0.34	28	D-
declared_close_to_use	-0.26	33	D-

Worst offender (lib/codeqa/metrics/file/identifier_length_variance.ex:39-44):

    %{
      "mean" => Float.round(mean, 4),
      "variance" => Float.round(variance, 4),
      "std_dev" => Float.round(std_dev, 4),
      "max" => lengths |> Enum.max()
    }

🟡 Testing — B+ (83/100)

Cosine similarity scores for 2 behaviors.

Behavior	Cosine	Score	Grade
test_single_concept	0.27	74	B
test_name_describes_behavior	0.53	91	A-

🟢 Type And Value — A- (90/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
hardcoded_url_or_path	0.49	90	A-

🟡 Variable Naming — B (74/100)

Cosine similarity scores for 1 behaviors.

Behavior	Cosine	Score	Grade
name_is_generic	0.25	74	B

Worst offender (lib/codeqa/block_impact/codebase_impact.ex:14-22):

  """
  @spec compute(String.t(), String.t(), Node.t(), map()) :: map()
  def compute(path, content, node, files_map) do
    root_tokens = TokenNormalizer.normalize_structural(content)
    reconstructed = FileImpact.reconstruct_without(root_tokens, node)
    updated_files = Map.put(files_map, path, reconstructed)
    Analyzer.analyze_codebase_aggregate(updated_files)
  end
end

github-actions · 2026-06-06T13:56:03Z

🟡 9 blocks with minor issues (safe to merge)

9 medium

9 medium-severity blocks (expand)

🟡 lib/codeqa/block_impact/codebase_impact.ex:14-22 — Name Contains Type Suffix

Issues:

🟡 MEDIUM variable_naming/name_contains_type_suffix (Δ 0.24)

  14 │   """
  15 │   @spec compute(String.t(), String.t(), Node.t(), map()) :: map()
  16 │   def compute(path, content, node, files_map) do
  17 │     root_tokens = TokenNormalizer.normalize_structural(content)
  18 │     reconstructed = FileImpact.reconstruct_without(root_tokens, node)
  19 │     updated_files = Map.put(files_map, path, reconstructed)
  20 │     Analyzer.analyze_codebase_aggregate(updated_files)
  21 │   end
  22 │ end

🟡 lib/codeqa/metrics/post_processing/post_processing_metric.ex:19-21 — Name Contains Type Suffix

Issues:

🟡 MEDIUM variable_naming/name_contains_type_suffix (Δ 0.23)

  19 │   """
  20 │   @callback analyze(pipeline_result :: map(), files_map :: map(), opts :: keyword()) :: map()
  21 │ end

🟡 lib/codeqa/analysis/run_context.ex:7-15 — Name Contains Type Suffix

Issues:

🟡 MEDIUM variable_naming/name_contains_type_suffix (Δ 0.22)

   7 │   """
   8 │ 
   9 │   defstruct [:behavior_config_pid, :file_context_pid]
  10 │ 
  11 │   @type t :: %__MODULE__{
  12 │           behavior_config_pid: pid(),
  13 │           file_context_pid: pid()
  14 │         }
  15 │ end

🟡 lib/codeqa/ast/nodes/shared.ex:11-27 — Name Contains Type Suffix

Issues:

🟡 MEDIUM variable_naming/name_contains_type_suffix (Δ 0.20)

  11 │   """
  12 │ 
  13 │   alias CodeQA.AST.Enrichment.Node
  14 │ 
  15 │   @spec cast_shared(module(), Node.t()) :: struct()
  16 │   def cast_shared(target_struct, %Node{} = node),
  17 │     do:
  18 │       target_struct
  19 │       |> struct(
  20 │         tokens: node.tokens,
  21 │         line_count: node.line_count,
  22 │         children: node.children,
  23 │         start_line: node.start_line,
  24 │         end_line: node.end_line,
  25 │         label: node.label
  26 │       )
  27 │ end

🟡 lib/codeqa/ast/classification/node_type_detector.ex:15-19 — Name Contains Type Suffix

Issues:

🟡 MEDIUM variable_naming/name_contains_type_suffix (Δ 0.20)

  15 │   """
  16 │   @spec detect_types([Node.t()], module()) :: [term()]
  17 │   def detect_types(blocks, lang_mod),
  18 │     do: blocks |> Enum.map(&NodeClassifier.classify(&1, lang_mod))
  19 │ end

🟡 lib/codeqa/metrics/codebase/codebase_metric.ex:1-10 — Name Contains Type Suffix

Issues:

🟡 MEDIUM variable_naming/name_contains_type_suffix (Δ 0.16)

   1 │ defmodule CodeQA.Metrics.Codebase.CodebaseMetric do
   2 │   @moduledoc """
   3 │   Behaviour for metrics that operate across an entire codebase.
   4 │ 
   5 │   Unlike `FileMetric`, which analyzes a single file, codebase metrics receive
   6 │   a map of all source files and can compute cross-file properties such as
   7 │   duplication or structural similarity.
   8 │ 
   9 │   See [software metrics](https://en.wikipedia.org/wiki/Software_metric).
  10 │   """

🟡 lib/codeqa/metrics/file/identifier_length_variance.ex:39-44 — Variable used only once

Issues:

🟡 MEDIUM scope_and_assignment/used_only_once (Δ 0.15)

  39 │     %{
  40 │       "mean" => Float.round(mean, 4),
  41 │       "variance" => Float.round(variance, 4),
  42 │       "std_dev" => Float.round(std_dev, 4),
  43 │       "max" => lengths |> Enum.max()
  44 │     }

🟡 lib/codeqa/ast/signals/structural/colon_indent_signal.ex:1-17 — Variable used only once

Issues:

🟡 MEDIUM scope_and_assignment/used_only_once (Δ 0.15)

   1 │ defmodule CodeQA.AST.Signals.Structural.ColonIndentSignal do
   2 │   alias CodeQA.AST.Lexing.NewlineToken
   3 │   alias CodeQA.AST.Lexing.WhitespaceToken
   4 │ 
   5 │   @moduledoc """
   6 │   Emits `:colon_indent_enclosure` for colon-indented blocks (Python).
   7 │ 
   8 │   Only active when `opts[:language_module]` returns true for `uses_colon_indent?/0`. Replaces
   9 │   `ParseRules.ColonIndentationRule`.
  10 │ 
  11 │   ## Limitation
  12 │ 
  13 │   The original rule flushes open blocks at EOF via `close_all_open/1`. Since
  14 │   `emit/3` has no end-of-stream callback, open blocks are instead flushed at
  15 │   each `<NL>` token. This correctly handles single-statement blocks; multi-line
  16 │   blocks are closed at the first newline (conservative).
  17 │   """

🟡 lib/codeqa/ast/parsing/signal_stream.ex:1-20 — Boolean function missing ? suffix

Issues:

🟡 MEDIUM function_design/boolean_function_has_question_mark (Δ 0.11)

   1 │ defmodule CodeQA.AST.Parsing.SignalStream do
   2 │   @moduledoc """
   3 │   Runs a list of `Signal` implementations over a token stream.
   4 │ 
   5 │   Each signal receives its own full pass over the token stream and accumulates
   6 │   its own state. Signals are independent — no shared state, no cross-signal
   7 │   coordination.
   8 │ 
   9 │   ## Return value
  10 │ 
  11 │   Returns a list of emission lists, one per signal, in the same order as the
  12 │   input signal list. Each emission is a 4-tuple:
  13 │ 
  14 │       {source, group, name, value}
  15 │ 
  16 │   ## Usage
  17 │ 
  18 │       SignalStream.run(tokens, [%BlankLineSignal{}, %KeywordSignal{}], [])
  19 │       # => [[{BlankLineSignal, :split, :blank_split, 5}, ...], [...]]
  20 │   """

aspala added 2 commits June 6, 2026 15:32

aspala merged commit 9da53a7 into main Jun 6, 2026
8 checks passed

aspala deleted the fix/separator-counts-loo branch June 6, 2026 13:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(metrics) + perf(health-report): correct separator LOO, scope block-impact to changed files#58

fix(metrics) + perf(health-report): correct separator LOO, scope block-impact to changed files#58
aspala merged 2 commits into
mainfrom
fix/separator-counts-loo

aspala commented Jun 6, 2026

Uh oh!

github-actions Bot commented Jun 6, 2026

Uh oh!

github-actions Bot commented Jun 6, 2026

Uh oh!

github-actions Bot commented Jun 6, 2026

🟡 9 blocks with minor issues (safe to merge)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

aspala commented Jun 6, 2026

Why

1. fix(metrics): remove incorrect separator_counts subtractive LOO

2. perf(health-report): scope per-node block impact to changed files

Verification

Uh oh!

github-actions Bot commented Jun 6, 2026

🟠 Code Health: C+ (65/100)

Uh oh!

github-actions Bot commented Jun 6, 2026

Uh oh!

github-actions Bot commented Jun 6, 2026

🟡 9 blocks with minor issues (safe to merge)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

1. fix(metrics): remove incorrect `separator_counts` subtractive LOO