MLOps Pathmlopspath.ru

05 · Observability5.3 · Трейсы, алертинг, SLO5.3.4сложный

Observability качества модели: drift и infra

Зачем это нужно

Prometheus зелёный: pods healthy, latency OK, errors 0.1%. Fraud team: «модель пропускает мошенников». Это не infra incident — это model quality degradation. Причины: data drift, concept drift, train-serve skew, wrong model version. Блок 5 замыкается разделением: infra observability vs model observability и как не путать их в triage.

Основные идеи

Infra observability (recap).

  • RED, USE, GPU, logs, traces, SLO on availability/latency.

  • Answers: «сервис жив и быстр?»

Model observability.

  • Answers: «predictions still valid for business?»

Data drift. Входное распределение P(X) изменилось vs training.

  • Example: средний tenure вырос после marketing campaign.

  • Detect: PSI, KS-test, compare histograms per feature, null rate alerts.

Concept drift. P(Y|X) изменилось при том же X.

  • Example: fraudsters adapted; old patterns fail.

  • Detect: rising false negatives in labeled feedback (delayed), proxy metrics (chargeback rate).

Train-serve skew. Разные преобразования offline vs online.

  • Example: training used StandardScaler fit on full data; serving forgets same scaler params.

  • Detect: schema tests, golden sample scores in CI, compare feature stats train vs prod logs (aggregated).

Prediction monitoring.

  • Distribution of scores (histogram over hour).

  • Rate of top-decile predictions.

  • Default/fallback path usage rate spike.

Label latency. True labels arrive days later — quality loops slower than infra loops. Design proxies and periodic batch evaluation.

Drift vs infra — triage table.

Signal Likely infra Likely model/data
5xx up ✓
p95 up, GPU idle dependency / network maybe heavy input
200 OK, business KPI down ✓
Constant prediction output bug / fallback ✓ drift or skew
After deploy only deploy/artifact ✓ skew or wrong weights
Gradual over weeks rare ✓ concept drift

Responses.

  • Infra: rollback, scale, fix pipeline (blocks 3–4).

  • Model: trigger retrain (Argo WF), rollback champion in MMS, feature fix upstream, human review before auto-promote.

Governance link. Auto-retrain without MMS approve — forbidden in corp contour; drift alert → ticket to DS, not silent prod swap.

Tools. Custom Grafana panels from batch jobs; Evidently AI / WhyLabs (concept); Great Expectations on streaming features; ClearML compare offline eval on fresh labeled sample.

Privacy. Monitoring aggregates only; no raw PII in drift dashboards.

Как это выглядит на практике

Weekly job drift-report-churn:

  1. Sample 10k production requests (aggregated features stored legally).

  2. Compare to training snapshot baseline.

  3. PSI > 0.2 on income → Slack #ml-quality warning.

  4. DS investigates upstream ETL change — not inference pod restart.

Live dashboard row «Model quality»:

  • Score distribution overlay train vs today.

  • % requests using fallback.

  • model_version label from MMS.

Incident merged: infra on-call + DS joint bridge.

  • Metrics: all green.

  • Drift panel: null rate on region 40% (was 0%).

  • Root: Feature API config typo after unrelated deploy.

  • Fix: feature team; inference rollback not needed but MMS documents incident on model card.

Capstone connection: E2E contour (4.4) + observability (block 5) = detect, mitigate, retrain with gates.

Что сделать после занятия

  • Перечислите 3 proxy-метрики качества для учебной модели без immediate labels.

  • Заполните triage table для fictional case «CTR упал, infra green».

  • Опишите process: drift alert → who acts → MMS/champion change or retrain.

Официальные материалы