Fix multiclass / non-{0,1}-label crashes across the sklearn classification tests (ZD-704)

[johnwalz97]

Pull Request Description

What and why?

A customer (ZD-704) running a multiclass model hit ValueErrors in the scikit-learn tests, one at a time. Rather than fix them reactively, I swept every classification sklearn test against a multiclass model and fixed all the genuine breakages.

Root cause (shared): these tests score classification metrics / build plots using scikit-learn defaults that only hold for a standard {0, 1} binary problem.

Tests fixed

Per-region diagnosis tests — WeakspotsDiagnosis, OverfitDiagnosis, RobustnessDiagnosis. They score precision/recall/F1/AUC on subsets of the data (feature bins, noise-perturbed copies):

• precision/recall/F1 default to average="binary", pos_label=1 → Target is multiclass but average='binary' (multiclass) and pos_label=1 is not a valid label (binary encoded outside {0,1}, e.g. {0,4}).
• roc_auc_score (default auc metric) → multi_class must be in ('ovo', 'ovr') on multiclass. A true probability-based multiclass ROC AUC isn't possible here (the library retains only a single probability column), so multiclass AUC falls back to the label-binarize convention already used by ClassifierPerformance.

The customer's WeakspotsDiagnosis case was a genuinely multiclass target where one feature bin happened to hold only two classes (0 and 4) — incidental, not a deliberate encoding (customer confirmed: not binary, not intentionally 0/4).

CalibrationCurve — wraps sklearn's binary-only calibration_curve, which raised a cryptic pos_label is not specified on multiclass. Now skips cleanly with SkipTestError, matching ROCCurve/PrecisionRecallCurve.

SHAPGlobalImportance — select_shap_values only handled the older SHAP list layout (one 2D array per class). Newer SHAP returns a single (samples, features, classes) array for multiclass, which fell through to the regression branch and was passed whole to shap.summary_plot → IndexError: index N is out of bounds for axis 1. Now selects along the trailing class axis; multiclass works with class_of_interest, and raises a clear, actionable error without it.

What changed

New shared module _diagnosis_metrics.py (resolve_averaging/full_labels, bind_averaging/apply_averaging, multiclass_auc) used by all three diagnosis tests. WeakspotsDiagnosis refactored onto it; OverfitDiagnosis/RobustnessDiagnosis route auc (multiclass) and label metrics through it. Averaging is resolved once from the full label set (union of true + predicted labels) so every subset is scored consistently; standard {0,1} stays on pos_label=1 (unchanged). Plus the two standalone fixes above.

Full multiclass sweep (21 classification tests)

Status	Tests
✅ Fixed here	WeakspotsDiagnosis, OverfitDiagnosis, RobustnessDiagnosis, CalibrationCurve (skips), SHAPGlobalImportance
✅ Already worked	ClassifierPerformance, ConfusionMatrix, MinimumAccuracy, MinimumF1Score, MinimumROCAUCScore, ModelParameters, ModelsPerformanceComparison, FeatureImportance, PermutationFeatureImportance, PopulationStabilityIndex, TrainingTestDegradation, HyperParametersTuning
✅ Binary-by-design, skip gracefully	ROCCurve, PrecisionRecallCurve, ClassifierThresholdOptimization

ScoreProbabilityAlignment needs a scorecard "score" column (not a multiclass issue). Clustering/regression tests aren't applicable to a classifier.

How to test

uv run python -m pytest tests/unit_tests/model_validation/sklearn/ -q

New/updated: shared-helper units (incl. multiclass_auc and select_shap_values edge cases) and end-to-end runs across binary {0,1}, binary {0,4}, multiclass (default and per-metric), and regression. Full sklearn unit dir: 66 passed.

What needs special review?

• Multiclass AUC semantics: no per-class probabilities are stored, so multiclass auc in Overfit/Robustness is a label-binarize AUC (matches ClassifierPerformance), not a probability ROC AUC.
• Resolving averaging globally (from the full dataset) rather than per subset — deliberate, so metrics stay comparable and sparse subsets don't flip to binary.
• pos_label = max(labels) for non-{0,1} binary targets.
• SHAPGlobalImportance now requires class_of_interest for multiclass (clear error if omitted) — consistent with the pre-existing list-branch contract, but confirm that's the desired UX vs. defaulting to a class.

Dependencies, breaking changes, and deployment notes

No breaking changes; {0,1} binary and regression results unchanged. Needs a new library release tagged to PyPI before the customer can pip install the fix.

Release notes

Fixed ValueErrors when running scikit-learn tests on multiclass targets (or binary targets encoded outside {0, 1}): WeakspotsDiagnosis, OverfitDiagnosis, and RobustnessDiagnosis now handle multiclass; CalibrationCurve skips cleanly for multiclass; and SHAPGlobalImportance supports multiclass via class_of_interest. Previously these raised errors such as Target is multiclass but average='binary', pos_label=1 is not a valid label, multi_class must be in ('ovo', 'ovr'), or a SHAP IndexError.

Checklist

• What and why
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied (bug)
• PR linked to Shortcut
• Unit tests added
• Tested locally
• Documentation updated (if required)

🤖 Generated with Claude Code

[github-actions]

PR Summary

This pull request significantly enhances the model validation module by introducing a broad suite of unit tests and a new helper module for diagnosis metrics. The changes include:

• New unit tests for various diagnosis functions in the sklearn model validation submodules such as CalibrationCurve, OverfitDiagnosis, RobustnessDiagnosis, SHAPGlobalImportance, WeakspotsDiagnosis, and diagnosis metrics. These tests cover both binary and multiclass scenarios, including edge cases like non-standard label encodings and sparse class slices.

• Introduction of a new helper module (_diagnosis_metrics.py) that encapsulates shared metric logic. This module provides functions to:

  • Compute the full set of labels across datasets
  • Resolve averaging strategies (returning a tuple of average type and pos_label) depending on whether the target is binary or multiclass
  • Bind averaging parameters (average, pos_label, zero_division) to metric functions in a reusable way
  • Compute a multiclass ROC AUC using label binarization when only a single probability column is available

• Modifications in the diagnosis implementations (OverfitDiagnosis, RobustnessDiagnosis, SHAPGlobalImportance, and WeakspotsDiagnosis) to consistently integrate the new averaging logic. This includes proper resolution of label spaces and binding of metric parameters to account for binary targets encoded outside the conventional {0, 1} or multiclass predictions.

These changes not only expand the test coverage to ensure robust functionality across a variety of scenarios but also improve the maintainability of the code by centralizing common diagnostic calculations and parameter bindings.

Test Suggestions

• Run all unit tests to ensure that existing functionality has not regressed after integrating the new helper module.
• Test additional edge cases with datasets having unusual label encodings (e.g., labels not in {0, 1}).
• Validate error handling by intentionally providing invalid parameters to functions like select_shap_values and verifying that clear and informative errors are raised.
• Perform integration tests to ensure that the diagnosis functions correctly handle both binary and multiclass datasets during real-world scenarios.
• Add performance tests to ensure that the added averaging and label resolution logic does not introduce significant overhead for large datasets.

[hunner]

Approving — this subsumes #529's follow-up work in #533 (which I'm closing in favor of this PR) and I verified the claims empirically rather than by read-through: a 15-scenario matrix (Weakspots / Overfit / Robustness × default auc / explicit f1 × labels {0,2,4} / {0,4} / {0,1}) run identically against main, #533, and this branch. main crashes on 9 of 15; this branch passes all 15, including the multiclass default-auc path that #533 left out of scope (and which would very likely have been this customer's next ticket). multiclass_auc does match ClassifierPerformance's LabelBinarizer convention as claimed, with sensible hardening on top (global-label fit for column alignment, scorable-class filtering; unseen perturbed labels binarize to all-zero rows rather than crashing). Local suite: 58 passed, 2 pre-existing skips (SHAP tests covered by the max-deps CI matrix, which is green).

Two findings worth fixing before merge, both demonstrated:

1. CalibrationCurve still crashes on binary targets encoded outside {0, 1} — the exact error this PR calls out as cryptic. The new guard only skips when len(np.unique(dataset.y)) > 2, so a two-class {0, 4} target sails past it and hits sklearn's ValueError: y_true takes value in {0, 4} and pos_label is not specified.... Reproduced on this branch with a {0, 4} logistic model. Since the PR's own convention (in resolve_averaging) is "largest label is the positive class", passing pos_label through to calibration_curve for the two-class case closes it consistently:

prob_true, prob_pred = calibration_curve(
    dataset.y, dataset.y_prob(model), n_bins=n_bins,
    pos_label=np.unique(dataset.y).max(),
)

2. apply_averaging in WeakspotsDiagnosis rebinds user-supplied custom metrics, silently overriding their explicit choices. bind_averaging rebinds any callable whose signature exposes average — and inspect.signature of functools.partial(f1_score, average="weighted") still exposes average, so the most natural way for a user to customize averaging gets clobbered. Demonstrated on this branch:

p = functools.partial(f1_score, average="weighted")
bind_averaging(p, "macro", None)(y_true, y_pred)
# returns the macro score (0.8413), not the user's weighted score (0.8492)

Since WeakspotsDiagnosis applies it to the whole prepared registry, this hits every custom metric, not just defaults. Suggest only applying averaging when the caller didn't pass metrics (i.e. rebind DEFAULT_METRICS only) — custom callables should own their kwargs.

Two non-blocking notes: the PR body's "every subset is scored consistently" slightly oversells — the strategy is resolved globally but macro without labels= still averages over each slice's present classes only (a {0,2} slice of a {0,2,4} problem: 0.80 here vs 0.53 if labels= were pinned). I actually think your behavior is the better choice for this test — a homogeneous, perfectly-predicted bin shouldn't be flagged as a weakspot — so just worth a precise sentence rather than a change. And the Shortcut checklist item: this is sc-17091.

🤖 Automated reply