5.2.3 · блок 5
Практика расследования инцидента
Практика расследования инцидента
Зачем это нужно
Теория metrics + logs + traces оживает в инциденте. ML-сервис «работает», но business жалуется — или наоборот, алерт красный, а пользователи не пострадали. Этот урок — пошаговый playbook расследования в стеке курса: Prometheus/Grafana → Loki → Istio → MMS/ClearML.
Цель — не паника и не random restart, а structured triage и blameless postmortem.
Основные идеи
Incident phases.
1. Detect — alert or ticket.
2. Triage — severity, customer impact, owner.
3. Mitigate — rollback, scale, traffic shift.
4. Diagnose — root cause.
5. Resolve — permanent fix.
6. Postmortem — learn, action items.
First 15 minutes checklist.
| Step | Tool | Question |
|------|------|----------|
| 1 | Status page / PM | Real user impact? |
| 2 | Grafana RED | Errors, latency, traffic change? |
| 3 | Argo CD | Last deploy? sync status? |
| 4 | MMS | Champion model version expected? |
| 5 | Loki | Error logs pattern since deploy? |
| 6 | HPA / pods | CrashLoop, OOM, Pending? |
| 7 | Istio | Canary weight correct? |
Symptom trees.
*High error rate*
- 5xx from app → logs stack trace → model load / dependency.
- 5xx only on new version → canary issue → rollback Istio + Argo.
- 4xx spike → client schema change / validation.
*High latency, low errors*
- GPU saturation → DCGM dashboard.
- Feature API slow → traces to dependency.
- Cold start / scale-up → check pod age and HPA events.
*«Silent» bad predictions*
- Infra green → model quality path (урок 5.3.4): drift, wrong artifact, train-serve skew.
Mitigation before root cause. Rollback to last known good если business impact high и MMS has previous Approved version. Parallel investigation.
Communication. Incident channel: status, hypothesis, ETA. Не «мы рестартуем pod» без контекста.
Evidence preservation. Screenshot dashboard time range; export relevant logs; Argo revision number; ClearML task id — для postmortem.
Blameless postmortem template.
- Timeline (UTC).
- Impact (users, revenue proxy).
- Root cause (technical + process).
- What went well / poorly.
- Action items (owner, due date): code, alert, runbook, gate.
Как это выглядит на практике
Scenario A: Deploy churn v1.3.0, error rate 15%
1. Alert fires: ChurnErrorRate > 1% for 5m.
2. Grafana: errors started at 14:02; deploy 14:00 (Argo).
3. Loki: ImportError sklearn in new pods.
4. Mitigate: Argo rollback to revision 41; Istio 100% v1.2.2.
5. RCA: promote Jenkins used Dockerfile without updated requirements.lock.
6. Action: CI gate pip check + import model in test image.
Scenario B: Latency p95 2s, errors 0.1%
1. RED: duration up, errors OK.
2. GPU util 20% — not GPU bound.
3. Traces (5.3.1): 1.8s in feature-store-grpc.
4. Mitigate: scale Feature Store, temporary cache in inference.
5. RCA: upstream deploy removed index on lookup table.
Scenario C: No alerts, fraud team says «model dumb»
1. Infra metrics normal.
2. Check /health model_version matches MMS.
3. Compare prediction distribution vs last week — shift detected.
4. RCA: data pipeline sent nulls for key feature; model imputes poorly.
5. Action: data quality alert on null rate; block promote in train pipeline.
Tabletop for команды: раздать scenario C junior on-call — оценить, не застрял ли только на RED.
Что сделать после занятия
- [ ] Напишите runbook на 1 страницу для «error rate high» с вашими tool URLs.
- [ ] Пройдите tabletop: scenario B — какие 5 queries вы выполните.
- [ ] Шаблон postmortem (5 секций) — заполните на учебный fictional incident.