feat(metrics): Rényi entropy spectrum & Hill numbers (#46)#57
Conversation
Add CodeQA.Metrics.File.RenyiEntropy: generalizes Shannon entropy across
order parameter α ∈ {0,1,2,∞} over the token distribution, yielding a
spectrum H₀ ≥ H₁ ≥ H₂ ≥ H_∞ that characterizes distribution shape rather
than a single scalar.
- renyi_0 (Hartley), renyi_1 (Shannon limit), renyi_2 (collision),
renyi_inf (min-entropy)
- hill_1, hill_2 = 2^H_α — effective vocabulary size
- spectrum_slope = H₀ − H₂ — concentration indicator
- edge cases: empty → zeros; single token → H=0, Hill=1.0
- registered after LexicalConcentration (complementary concentration measure)
🟠 Code Health: C+ (63/100)
%%{init: {'theme': 'neutral'}}%%
xychart-beta
title "Code Health Scores"
x-axis ["Readability", "Complexity", "Structure", "Duplication", "Naming", "Magic Numbers", "Combined Metrics"]
y-axis "Score" 0 --> 100
bar [94, 31, 88, 48, 96, 100, 65]
|
🔍 Top Likely Issues (cosine similarity)
🟢 Readability — A (94/100)Codebase averages: flesch_adapted=97.85, fog_adapted=4.81, avg_tokens_per_line=9.50, avg_line_length=35.68
🔴 Complexity — D- (31/100)Codebase averages: difficulty=41.03, effort=231306.84, volume=4013.14, estimated_bugs=1.34
🟢 Structure — A- (88/100)Codebase averages: branching_density=0.14, mean_depth=3.86, avg_function_lines=8.26, max_depth=9.20, max_function_lines=19.85, variance=6.86, avg_param_count=1.15, max_param_count=2.04
🟠 Duplication — C- (48/100)Codebase averages: redundancy=0.59, bigram_repetition_rate=0.54, trigram_repetition_rate=0.37
🟢 Naming — A (96/100)Codebase averages: entropy=0.89, mean=6.65, variance=18.81, avg_sub_words_per_id=1.17
🟢 Magic Numbers — A (100/100)Codebase averages: density=0.00
🔴 Combined Metrics — D (65/100)
🔴 Code Smells — D- (25/100)
🟡 Consistency — B+ (81/100)
🔴 Dependencies — E+ (19/100)
🟡 Documentation — B+ (83/100)
🟢 Error Handling — A- (92/100)
🟠 File Structure — C- (48/100)
🟡 Function Design — B+ (81/100)
🟢 Naming Conventions — A- (90/100)
🔴 Scope And Assignment — D- (28/100)
🟡 Testing — B+ (83/100)
🟢 Type And Value — A- (89/100)
🟡 Variable Naming — B (74/100)
|
kind: refactoring-tasks
|
# Conflicts: # priv/combined_metrics/code_smells.yml # priv/combined_metrics/function_design.yml
Was & Warum
Schließt #46.
Metrics.File.Entropyliefert nur Shannon (α=1). Rényi-Entropie verallgemeinert das über einen Order-Parameter α: kleine α gewichten seltene Tokens, große α nur die häufigsten. Ergebnis ist ein Spektrum statt eines Skalars — charakterisiert die Verteilungsform robuster.Hill-Numbers
D_α = 2^(H_α)sind die „effektive Vokabulargröße": wie viele Tokens wirklich zählen — interpretierbarer als rohevocab_size.Frequenzen kommen aus
ctx.token_counts, also quasi gratis (gleicher Aufbau wie Shannon).Umsetzung
Neues Modul
lib/codeqa/metrics/file/renyi_entropy.ex,@behaviour FileMetric. 7 keys:renyi_0log2(vocab_size)— Hartley, zählt nur Supportrenyi_1-Σ pᵢ log2 pᵢ— Shannon-Limit (α→1)renyi_2-log2(Σ pᵢ²)— Collisionrenyi_inf-log2(max pᵢ)— Min-Entropiehill_1/hill_22^H_α— effektive Vokabulargrößespectrum_slopeH₀ − H₂— Konzentrations-Indikator (roh, wie spezifiziert)analyzer.exdirekt nachLexicalConcentration— beide messen Konzentration (Rényi über Entropy-Achse, Yule/Simpson über Frequenz-Spektrum), komplementär (feat(metrics): Yule's K / Simpson's D (längen-invariante Vokabular-Konzentration) #44).Float.round(_, 4)— konsistent mit den neueren Metric-Modulen.H = 0,hill = 1.0(= vocab_size).analyze_loobewusst weggelassen: Entropie ist nichtlinear, also nicht trivial subtraktiv ausbaseline. Block-Analyzer fällt sauber auf full re-run zurück (Fallback viafunction_exported?).Test
test/codeqa/metrics/file/renyi_entropy_test.exs, Muster wiebrevity_test.exs. 13 Tests:a b c d e f g h):renyi_0 == renyi_2,spectrum_slope == 0, alle Orders kollabieren,hill == vocab_size == 8spectrum_slope, MonotonieH₀ ≥ H₁ ≥ H₂ ≥ H_∞,hill_2 << vocab_sizekeys/0deckt sich mitanalyze/1-OutputVerify
mix test— 904 Tests, 0 failuresmix compile --warnings-as-errorscleanmix format --check-formattedclean