ML Case Lab (Selection/CV/Leakage/PR-AUC/SHAP)

This lesson on ML Case Lab (Selection/CV/Leakage/PR-AUC/SHAP) is hands-on and example-driven. You will master framing and defending an end-to-end machine learning case study for churn and customer lifetime value (LTV). You will learn how to design leakage-free grouped cross-validation schemes, evaluate imbalanced predictions using PR-AUC and Brier scores, and translate SHAP feature attributions into business threshold decisions.

What You'll Be Able To Do

• Design a GroupKFold cross-validation strategy that isolates user IDs across folds to prevent target leakage.
• Evaluate imbalanced churn classification models using PR-AUC and Brier calibration curves rather than misleading ROC-AUC metrics.
• Calculate cost-optimal decision thresholds by trading off customer retention intervention cost against predicted lifetime value.
• Explain individual and global model predictions to executive stakeholders using SHAP TreeExplainer summary and force plots.

Detailed Concept Walkthrough

1. Grouped Cross-Validation and Leakage Prevention

Standard random splitting leaks user-level temporal signals and dependencies when multiple interaction records exist per customer. Grouped cross-validation ensures that all records belonging to a single entity remain strictly within either train or validation splits.

• Mechanism: GroupKFold partitions the dataset such that the distinct group identifier (e.g., user_id or account_id) is mutually exclusive across all validation folds, preventing intra-group memorization.
• Under the Hood: Cross-validation splitters build index arrays based on unique group hashes, guaranteeing zero intersection of group keys while attempting to maintain balanced sample counts per fold.
• Best Practice: Always apply all feature transformations and imputations inside the cross-validation loop via scikit-learn Pipelines to avoid global statistical leakage.

from sklearn.model_selection import GroupKFold
import numpy as np

gkf = GroupKFold(n_splits=5)
# X: features, y: churn_label, groups: user_id array
for fold, (train_idx, val_idx) in enumerate(gkf.split(X, y, groups=df['user_id'])):
    X_tr, y_tr = X.iloc[train_idx], y.iloc[train_idx]
    X_va, y_val = X.iloc[val_idx], y.iloc[val_idx]
    # Train and evaluate fold without leakage
    print(f"Fold {fold} - Train users: {len(np.unique(df['user_id'].iloc[train_idx]))}")

Key Takeaway: Never split repeated-measurement data randomly; group by user identity to measure true generalization.

2. Evaluation with PR-AUC and Brier Score

ROC-AUC masks poor minority-class precision in highly imbalanced churn scenarios due to a large true negative count. PR-AUC and Brier score assess positive class ranking precision and probability calibration directly.

• Mechanism: PR-AUC integrates precision over recall levels, ignoring true negatives and spotlighting false positive rates among rare churn events.
• Under the Hood: Brier score computes the mean squared difference between predicted probabilities and actual binary outcomes, serving as a strictly proper scoring rule for probability calibration.
• Best Practice: Pair average precision score with probability calibration curves to verify that a predicted 0.7 probability corresponds to a true 70% churn rate.

from sklearn.metrics import precision_recall_curve, auc, brier_score_loss

# y_prob contains continuous probabilities for class 1 (churn)
precision, recall, thresholds = precision_recall_curve(y_true, y_prob)
pr_auc_score = auc(recall, precision)
brier = brier_score_loss(y_true, y_prob)

print(f"PR-AUC: {pr_auc_score:.4f} | Brier Score: {brier:.4f}")

Key Takeaway: Use PR-AUC to assess rare-event ranking and Brier score to ensure probabilities reflect real-world risk.

3. Optimal Thresholding with Business Cost Curves

The default 0.5 classification threshold is arbitrary and ignores the asymmetric business economics of false positives versus false negatives. Threshold optimization maps model probabilities to expected profit or retained LTV.

• Mechanism: Sweeping candidate thresholds evaluates total utility: Utility = (TP * Net_Benefit) - (FP * Intervention_Cost) - (FN * Lost_LTV).
• Under the Hood: Cost-utility curves identify the probability cutoff where marginal customer retention value equals the marginal cost of preventative customer outreach.
• Best Practice: Report the operating threshold alongside the estimated annualized dollar return rather than raw precision or recall percentages in interview cases.

import numpy as np

def find_optimal_threshold(y_true, y_prob, cost_fp, benefit_tp, ltv_fn):
    thresholds = np.linspace(0.01, 0.99, 100)
    utilities = []
    for t in thresholds:
        preds = (y_prob >= t).astype(int)
        tp = np.sum((preds == 1) & (y_true == 1))
        fp = np.sum((preds == 1) & (y_true == 0))
        fn = np.sum((preds == 0) & (y_true == 1))
        utility = (tp * benefit_tp) - (fp * cost_fp) - (fn * ltv_fn)
        utilities.append(utility)
    return thresholds[np.argmax(utilities)]

Key Takeaway: Select operational thresholds based on maximal dollar utility rather than an arbitrary 0.5 probability.

4. Explainability and Storytelling with SHAP

SHAP (SHapley Additive exPlanations) attributes a prediction to individual feature contributions based on cooperative game theory. It bridges the gap between black-box tree ensembles and actionable stakeholder storytelling.

• Mechanism: TreeExplainer calculates exact Shapley values in polynomial time, allocating the deviation of a prediction from the baseline expected value to each feature.
• Under the Hood: Global feature importance aggregates mean absolute SHAP values across all rows, whereas local waterfall plots explain why a single user is at risk.
• Best Practice: Use SHAP summary beeswarm plots to show both feature magnitude and directionality when defending feature engineering choices.

import shap
import xgboost as xgb

model = xgb.XGBClassifier().fit(X_tr, y_tr)
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_va)

# Summary plot for global interpretability
shap.summary_plot(shap_values, X_va)
# Waterfall plot for an individual churn case
shap.plots.waterfall(shap_values[0])

Key Takeaway: Translate SHAP attributions into concrete operational levers to make model outputs actionable for product teams.

Topics Covered in ML Case Lab (Selection/CV/Leakage/PR-AUC/SHAP)

1. Case Setup & Grouped CV (0:00 - 2:30) — Framing churn prediction requirements and establishing non-leaking grouped cross-validation strategies.
2. PR-AUC vs ROC-AUC (2:30 - 5:15) — Demonstrating how class imbalance distorts ROC metrics and why PR-AUC reveals true positive class performance.
3. Brier Score & Calibration (5:15 - 7:45) — Evaluating predicted probability reliability to ensure risk scores map directly to real-world outcomes.
4. Cost-Optimal Thresholding (7:45 - 10:15) — Mapping model decision boundaries to retention campaign costs and customer lifetime value savings.
5. SHAP Storytelling (10:15 - 12:00) — Translating complex tree ensemble outputs into actionable stakeholder narratives with SHAP plots.

DS Interview Prep Cheat Sheet

• GroupKFold(n_splits=k) — Splits data ensuring groups do not overlap across folds

  GroupKFold(n_splits=5).split(X, y, groups=df['user_id'])
  

• precision_recall_curve(y_true, y_score) — Computes precision-recall pairs across varying probability thresholds

  p, r, t = precision_recall_curve(y_val, y_pred_prob)
  

• brier_score_loss(y_true, y_prob) — Calculates mean squared error of predicted probabilities

  brier = brier_score_loss(y_val, y_pred_prob)
  

• shap.TreeExplainer(model) — Fast tree-specific Shapley attribution calculation engine

  explainer = shap.TreeExplainer(model)
  

• shap.summary_plot(shap_values, X) — Displays global feature impact, directionality, and dispersion

  shap.summary_plot(shap_values, X_val)
  

• shap.plots.waterfall(shap_values[i]) — Visualizes step-by-step feature contributions for single instance

  shap.plots.waterfall(shap_values[0])
  

Comparison Table

Common Pitfalls

• Mistake: Splitting customer interaction data randomly instead of grouping by user ID. Avoid: GroupKFold cross-validation partitioned strictly by user or account identifiers.
• Mistake: Relying on ROC-AUC for churn models with single-digit positive rates. Avoid: PR-AUC and precision-recall tradeoff curves to expose false alarm rates.
• Mistake: Deploying models with default 0.5 decision thresholds without evaluating business costs. Avoid: Utility-cost matrices balancing intervention expense against saved customer LTV.
• Mistake: Explaining ensemble models using legacy Gini impurity importance metrics. Avoid: SHAP values that provide consistent directionality and instance-level attribution.

FAQs

• Why is ROC-AUC misleading on imbalanced datasets? ROC-AUC includes true negatives in its denominator (false positive rate). A huge pool of non-churners deflates the false positive rate, making a poorly performing model appear highly accurate.
• When should I use Brier score over PR-AUC? Use Brier score when the exact probability value determines business action (such as discounting tiered by risk), whereas PR-AUC is preferred when only relative ranking matters.
• How do SHAP TreeExplainer values differ from standard feature importances? Standard feature importance measures total split improvement without showing direction; SHAP values assign additive, signed contributions indicating whether a feature increased or decreased churn risk.