5.3.4 · блок 5
Observability качества модели: drift и infra
Observability качества модели: drift и infra
Зачем это нужно
Prometheus зелёный: pods healthy, latency OK, errors 0.1%. Fraud team: «модель пропускает мошенников». Это не infra incident — это model quality degradation. Причины: data drift, concept drift, train-serve skew, wrong model version. Блок 5 замыкается разделением: infra observability vs model observability и как не путать их в triage.
Основные идеи
Infra observability (recap).
- RED, USE, GPU, logs, traces, SLO on availability/latency.
- Answers: «сервис жив и быстр?»
Model observability.
- Answers: «predictions still valid for business?»
Data drift. Входное распределение P(X) изменилось vs training.
- Example: средний
tenureвырос после marketing campaign. - Detect: PSI, KS-test, compare histograms per feature, null rate alerts.
Concept drift. P(Y|X) изменилось при том же X.
- Example: fraudsters adapted; old patterns fail.
- Detect: rising false negatives in labeled feedback (delayed), proxy metrics (chargeback rate).
Train-serve skew. Разные преобразования offline vs online.
- Example: training used
StandardScalerfit on full data; serving forgets same scaler params. - Detect: schema tests, golden sample scores in CI, compare feature stats train vs prod logs (aggregated).
Prediction monitoring.
- Distribution of scores (histogram over hour).
- Rate of top-decile predictions.
- Default/fallback path usage rate spike.
Label latency. True labels arrive days later — quality loops slower than infra loops. Design proxies and periodic batch evaluation.
Drift vs infra — triage table.
| Signal | Likely infra | Likely model/data |
|--------|--------------|-------------------|
| 5xx up | ✓ | |
| p95 up, GPU idle | dependency / network | maybe heavy input |
| 200 OK, business KPI down | | ✓ |
| Constant prediction output | bug / fallback | ✓ drift or skew |
| After deploy only | deploy/artifact | ✓ skew or wrong weights |
| Gradual over weeks | rare | ✓ concept drift |
Responses.
- Infra: rollback, scale, fix pipeline (blocks 3–4).
- Model: trigger retrain (Argo WF), rollback champion in MMS, feature fix upstream, human review before auto-promote.
Governance link. Auto-retrain without MMS approve — forbidden in corp contour; drift alert → ticket to DS, not silent prod swap.
Tools. Custom Grafana panels from batch jobs; Evidently AI / WhyLabs (concept); Great Expectations on streaming features; ClearML compare offline eval on fresh labeled sample.
Privacy. Monitoring aggregates only; no raw PII in drift dashboards.
Как это выглядит на практике
Weekly job drift-report-churn:
1. Sample 10k production requests (aggregated features stored legally).
2. Compare to training snapshot baseline.
3. PSI > 0.2 on income → Slack #ml-quality warning.
4. DS investigates upstream ETL change — not inference pod restart.
Live dashboard row «Model quality»:
- Score distribution overlay train vs today.
% requests using fallback.model_versionlabel from MMS.
Incident merged: infra on-call + DS joint bridge.
- Metrics: all green.
- Drift panel: null rate on
region40% (was 0%). - Root: Feature API config typo after unrelated deploy.
- Fix: feature team; inference rollback not needed but MMS documents incident on model card.
Capstone connection: E2E contour (4.4) + observability (block 5) = detect, mitigate, retrain with gates.
Что сделать после занятия
- [ ] Перечислите 3 proxy-метрики качества для учебной модели без immediate labels.
- [ ] Заполните triage table для fictional case «CTR упал, infra green».
- [ ] Опишите process: drift alert → who acts → MMS/champion change or retrain.