Case 6 (Capstone) — Incident Response: Diagnose a Model That Worked Yesterday but Not Today

# Case 6 (Capstone) — Incident Response: Diagnose a Model That Worked Yesterday but Not Today > **Enterprise Scenario:** At 08:30 on Tuesday morning, executives and support teams escalate a critical production outage: an automated credit card approval model has rejected 88% of applicants over the last 4 hours (normal rejection rate is 18%). The web service reports 100% HTTP 200 uptime, CPU utilization is normal, and latency is healthy at 24ms. However, thousands of qualified applicants are being rejected, costing the company $80,000 per hour. As on-call MLOps engineer, you must execute the Incident Response Playbook: triage telemetry within 15 minutes, execute an immediate atomic rollback to restore business operations, perform forensic data analysis to isolate the silent root cause, and author a postmortem with preventative guardrails. --- ## Technical Requirements & Deliverables ### Deliverable 1: Rapid Incident Triage (`triage_incident.py`) - Query the production prediction log table without impacting live database operations. - Compute rolling hourly metrics: rejection percentage, mean predicted credit score, and volume. - Confirm the severity of the failure and document initial blast radius. ### Deliverable 2: Emergency Atomic Rollback Automation (`rollback_service.py`) - Execute programmatic rollback via the MLflow Model Registry API: - Transition the faulty model version (`v4`) from `Production` to `Archived`. - Promote the previous stable version (`v3`) to `Production`. - Provide a secondary fallback: an emergency heuristic rule-based bypass that routes requests safely if the registry is unreachable. ### Deliverable 3: Forensic Root-Cause Analysis (`isolate_bug.py`) - Analyze raw incoming feature payloads across the pre-incident and post-incident time windows. - Uncover the silent data pipeline bug: an upstream CRM update overnight altered the column header `customer_credit_tenure_months` to `credit_tenure_months`. The downstream feature pipeline received `NULL` for the expected column, silently coerced missing values to `0` via default imputation, making every applicant appear to have zero credit history. ### Deliverable 4: Blameless Postmortem & Preventative Hardening - Author a formal postmortem report covering: - Incident Summary & Executive Impact. - Comprehensive Timeline of Events. - Root Cause Analysis (The 5 Whys). - Where Detection Succeeded and Failed (Why HTTP health checks passed). - Implement two code guardrails: 1. Strict Pydantic input contract asserting non-null constraints on critical credit features. 2. Production Circuit Breaker: A real-time statistical monitor that triggers an automated rollback if rejection rates exceed 35% across any 15-minute rolling window. --- ## Starter Code Scaffold ```python import mlflow from mlflow.tracking import MlflowClient import pandas as pd client = MlflowClient() def rollback_model_version(model_name: str = "credit_approval_model"): # Atomically rollback Production alias to previous stable version. prod_version = client.get_model_version_by_alias(model_name, "Production") prev_version = int(prod_version.version) - 1 print(f"Rolling back {model_name}: demoting v{prod_version.version}, promoting v{prev_version}") client.set_registered_model_alias(model_name, "Production", str(prev_version)) client.set_model_version_tag(model_name, str(prod_version.version), "incident_status", "quarantined") def forensic_null_audit(logs_df: pd.DataFrame, critical_cols: list[str]) -> dict: # Analyze null rates and zero-variance anomalies across production feature logs. null_rates = logs_df[critical_cols].isnull().mean() zero_rates = (logs_df[critical_cols] == 0).mean() return { "null_rates": null_rates.to_dict(), "zero_rates": zero_rates.to_dict() } ``` --- ## 100-Point Graded Rubric * **Criteria 1: Rapid Incident Triage & Impact Assessment (25 pts)** - *25 pts:* Formulates immediate quantitative assessment from production telemetry; confirms anomaly without modifying live state. - *15 pts:* Confirms issue but executes invasive live queries or modifies state prematurely. - *0 pts:* Fails to isolate the timeline or impact of the incident. * **Criteria 2: Atomic Rollback & Circuit Breakers (25 pts)** - *25 pts:* Executes clean programmatic Model Registry rollback in under 2 minutes; implements secondary rule-based fallback. - *15 pts:* Rollback requires manual container rebuild or lengthy code deploy. - *0 pts:* Cannot restore service; leaves failing model in production. * **Criteria 3: Forensic Root-Cause Isolation (25 pts)** - *25 pts:* Successfully uncovers silent schema drift (column rename coerced to default zero); proves why traditional software probes failed. - *15 pts:* Identifies that tenure is zero but does not trace upstream schema rename cause. - *0 pts:* Attributes failure incorrectly to model overfitting or random drift. * **Criteria 4: Blam