MLOps Pathmlopspath.ru

05 · Observability5.2 · Логи5.2.3средний

Практика расследования инцидента

Зачем это нужно

Теория metrics + logs + traces оживает в инциденте. ML-сервис «работает», но business жалуется — или наоборот, алерт красный, а пользователи не пострадали. Этот урок — пошаговый playbook расследования в стеке курса: Prometheus/Grafana → Loki → Istio → MMS/ClearML.

Цель — не паника и не random restart, а structured triage и blameless postmortem.

Основные идеи

Incident phases.

  1. Detect — alert or ticket.

  2. Triage — severity, customer impact, owner.

  3. Mitigate — rollback, scale, traffic shift.

  4. Diagnose — root cause.

  5. Resolve — permanent fix.

  6. Postmortem — learn, action items.

First 15 minutes checklist.

Step Tool Question
1 Status page / PM Real user impact?
2 Grafana RED Errors, latency, traffic change?
3 Argo CD Last deploy? sync status?
4 MMS Champion model version expected?
5 Loki Error logs pattern since deploy?
6 HPA / pods CrashLoop, OOM, Pending?
7 Istio Canary weight correct?

Symptom trees.

High error rate

  • 5xx from app → logs stack trace → model load / dependency.

  • 5xx only on new version → canary issue → rollback Istio + Argo.

  • 4xx spike → client schema change / validation.

High latency, low errors

  • GPU saturation → DCGM dashboard.

  • Feature API slow → traces to dependency.

  • Cold start / scale-up → check pod age and HPA events.

«Silent» bad predictions

  • Infra green → model quality path (урок 5.3.4): drift, wrong artifact, train-serve skew.

Mitigation before root cause. Rollback to last known good если business impact high и MMS has previous Approved version. Parallel investigation.

Communication. Incident channel: status, hypothesis, ETA. Не «мы рестартуем pod» без контекста.

Evidence preservation. Screenshot dashboard time range; export relevant logs; Argo revision number; ClearML task id — для postmortem.

Blameless postmortem template.

  • Timeline (UTC).

  • Impact (users, revenue proxy).

  • Root cause (technical + process).

  • What went well / poorly.

  • Action items (owner, due date): code, alert, runbook, gate.

Как это выглядит на практике

Scenario A: Deploy churn v1.3.0, error rate 15%

  1. Alert fires: ChurnErrorRate > 1% for 5m.

  2. Grafana: errors started at 14:02; deploy 14:00 (Argo).

  3. Loki: ImportError sklearn in new pods.

  4. Mitigate: Argo rollback to revision 41; Istio 100% v1.2.2.

  5. RCA: promote Jenkins used Dockerfile without updated requirements.lock.

  6. Action: CI gate pip check + import model in test image.

Scenario B: Latency p95 2s, errors 0.1%

  1. RED: duration up, errors OK.

  2. GPU util 20% — not GPU bound.

  3. Traces (5.3.1): 1.8s in feature-store-grpc.

  4. Mitigate: scale Feature Store, temporary cache in inference.

  5. RCA: upstream deploy removed index on lookup table.

Scenario C: No alerts, fraud team says «model dumb»

  1. Infra metrics normal.

  2. Check /health model_version matches MMS.

  3. Compare prediction distribution vs last week — shift detected.

  4. RCA: data pipeline sent nulls for key feature; model imputes poorly.

  5. Action: data quality alert on null rate; block promote in train pipeline.

Tabletop for команды: раздать scenario C junior on-call — оценить, не застрял ли только на RED.

Что сделать после занятия

  • Напишите runbook на 1 страницу для «error rate high» с вашими tool URLs.

  • Пройдите tabletop: scenario B — какие 5 queries вы выполните.

  • Шаблон postmortem (5 секций) — заполните на учебный fictional incident.

Официальные материалы