[SC-17128] ZD-717: Add multiclass (one-vs-rest) support to VM tests

[AnilSorathiya]

Pull Request Description

What and why?

Adds multiclass (one-vs-rest) support to four sklearn model-validation tests that were
previously binary-only, so customers validating multiclass classifiers get usable results
instead of a skipped/binary-only test (requested via ZD-717).

Per-class probabilities are read from the underlying estimator's predict_proba — the
VMModel wrapper only exposes the positive-class column, which is binary-only — and the true
labels are one-hot encoded with label_binarize in sklearn's sorted-class order to stay
aligned with the predict_proba columns.

• ROCCurve — for >2 classes, plots one OvR ROC curve per class plus a micro-average, each
  with its own AUC; RawData returns per-class fpr/tpr/auc keyed by class label plus a
  micro entry. Docstring updated for multiclass.
• PrecisionRecallCurve — per-class OvR precision-recall curves plus a micro-average.
• ConfusionMatrix — for >2 classes uses the model's argmax class predictions; the
  threshold parameter is binary-only and is ignored for multiclass.
• PopulationStabilityIndex — computes PSI one-vs-rest, one table/plot per class.

Binary behavior is unchanged in all four. Where a full per-class probability matrix isn't
available (metadata-only models, precomputed single-column probabilities, or a shape that
doesn't match the class count), the multiclass path raises a clear SkipTestError rather than
crashing.

How to test

Automated tests

Run the added/updated unit tests (this repo runs unittest via uv):

make test ONLY="tests.unit_tests.model_validation.sklearn.test_ROCCurve tests.unit_tests.model_validation.sklearn.test_PrecisionRecallCurve"

New multiclass coverage: TestROCCurveMulticlass.test_multiclass_one_vs_rest_traces and
TestPrecisionRecallCurveMulticlass.test_multiclass_one_vs_rest_traces (both train a 3-class
XGBoost model and assert one curve per class + a micro-average trace + baseline, and per-class
AUCs above 0.5). Binary tests in the same files should continue to pass.

Note: the multiclass tests require the xgboost optional extra and are skipped without it.

Manual testing

Run notebooks/code_samples/multiclass_classification_tests.ipynb end-to-end against a
3-class model and confirm:

• ROC and PR plots show one curve per class plus a micro-average curve.
• The confusion matrix renders an N×N grid from argmax predictions.
• PSI renders one table/plot per class.
• Re-running with a binary model produces the original single-curve output.

What needs special review?

• Label/column alignment: label_binarize(y, classes=...) must match predict_proba
  column order (sklearn's sorted class labels). Misalignment would silently mislabel per-class
  curves.
• Skip conditions: the SkipTestError fallbacks should skip only genuinely unsupported
  models (no per-class predict_proba, wrong-shaped probability matrix) and not falsely skip
  supported ones.
• ConfusionMatrix: threshold is now ignored for multiclass — confirm this is the desired
  behavior and is clearly documented.

Dependencies, breaking changes, and deployment notes

None. Library-only change, no migrations, no new env vars, no companion frontend/backend PR.
Binary output is unchanged, so no breaking changes.

Shortcut: SC-17128 ·

Release notes

ROC, Precision-Recall, Confusion Matrix, and Population Stability Index tests now support
multiclass classification models, in addition to binary. For models with more than two
classes, ROC and Precision-Recall curves are drawn one-vs-rest (one curve per class plus a
micro-average), the confusion matrix uses the model's predicted class, and PSI is reported per
class.

Checklist

• What and why
• Screenshots or videos (Frontend) — N/A (no UI change)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied — enhancement
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally — test_ROCCurve + test_PrecisionRecallCurve pass (5 tests, incl. multiclass OvR)
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[cachafla]

Tested the notebook and is working great 👏

If we're going to distribute this notebook I suggest:

• Ensuring it has the same structure as other public notebooks (intro text, code snippet, footer, etc.
• Put it inside a directory, right now it's at the root of code_samples/

Otherwise, we can keep it inside code_sharing/.

[AnilSorathiya]

Tested the notebook and is working great 👏

If we're going to distribute this notebook I suggest:

• Ensuring it has the same structure as other public notebooks (intro text, code snippet, footer, etc.
• Put it inside a directory, right now it's at the root of code_samples/

Otherwise, we can keep it inside code_sharing/.

It is not something for demonstration. We can keep in inside code_sharing.

[github-actions]

PR Summary

This PR introduces comprehensive enhancements and bug fixes to the model validation tests for sklearn classification models. The key changes focus on adding explicit multiclass handling for several metrics, including ROC Curve, Precision-Recall Curve, Confusion Matrix, and Population Stability Index.

The updates include:

• New or modified notebooks that exercise multiclass scenarios using one-vs-rest strategies. For ROC and Precision-Recall curves, the tests now generate one curve per class along with a micro-average. For the Confusion Matrix, the test now renders an N×N matrix for multiclass predictions, using the model’s argmax predictions (ignoring threshold parameters used for binary classification).

• Enhancements in the Population Stability Index implementation. The PSI now supports multiclass cases by computing one-vs-rest PSI for each class, adding detailed tables and plots for improved analysis. Helper functions have been added to format the PSI outputs, including appending a total row in the tables.

• Adjustments to precision-recall and ROC curve functions to ensure models that cannot produce a full per-class probability matrix (e.g. models with metadata-only predictions or single-column probabilities) are gracefully skipped rather than crashing, raising a SkipTestError.

• New unit tests for both binary and multiclass processes using XGBoost classifiers. The tests verify not only the correctness of the visual outputs (plotly Figures) but also the underlying raw metric values (e.g., average precision, AUC scores) to ensure that the one-vs-rest breakdown works as expected for multiclass scenarios.

Overall, this PR refines the behavior for classification model validations by accommodating multiclass support, ensuring graceful degradation for incompatible models, and confirming the outputs with robust unit tests.

Test Suggestions

• Run the updated notebooks to verify that both binary and multiclass models produce the correct set of curves and tables.
• Simulate a model with a single-column probability output to ensure that the SkipTestError is raised appropriately for multiclass metrics.
• Use the provided unit tests to validate ROC, Precision-Recall, and Population Stability Index outputs for both binary and multiclass cases.
• Manually inspect the generated plots to confirm that all expected traces (per-class and micro-average) are present.

[johnwalz97]

lgtm