MLOps Pathmlopspath.ru

05 · Observability5.3 · Трейсы, алертинг, SLO5.3.2сложный

Alerting: от метрики до действия

Зачем это нужно

Dashboard без алертов — музей. On-call просыпается, когда пользователи уже пострадали, если нет проактивных правил. Плохие алерты — «alert fatigue»: команда игнорирует всё. Хорошие алерты — actionable, привязаны к симптомам и runbook.

Для ML inference алертируем infra (RED, GPU) и позже quality (drift) — разные severity и каналы.

Основные идеи

Alert pipeline.

Prometheus/VM rule → Alertmanager → route → PagerDuty/Slack/email Grafana managed alerts → contact point (alternative path)

Prometheus alerting rule example.

` groups:

  • name: churn-serving rules:
    • alert: ChurnHighErrorRate expr: | sum(rate(istio_requests_total{destination_service="churn-serving",response_code=~"5.."}[5m])) / sum(rate(istio_requests_total{destination_service="churn-serving"}[5m]))

      0.05 for: 5m labels: severity: critical service: churn-serving annotations: summary: "Churn error rate above 5%" runbook_url: "https://wiki/runbooks/churn-errors"

`

for duration. Условие должно держаться N минут — фильтр flapping.

Severity levels.

Level Meaning Example
critical User-facing outage error rate > 5%, all pods down
warning Degraded, time to act p95 latency 2× SLO, GPU mem 90%
info FYI deploy completed

Alertmanager routing.

  • severity=critical → PagerDuty on-call.

  • severity=warning → Slack #ml-ops.

  • team=ds → drift alerts (5.3.4).

Actionable annotations. summary, description, runbook_url, dashboard_url. «Something is wrong» — плохо.

Symptom-based vs cause-based.

  • Symptom: high error rate, SLO burn — page on-call.

  • Cause: «disk 80%» — ticket unless causes outage soon.

ML-specific alerts.

  • Inference RED breaches.

  • All replicas not Ready after deploy (Argo + kube alerts).

  • GPU XID errors.

  • Prediction distribution shift (batch job alert — not always real-time page).

Silences and maintenance. Alertmanager silence during planned deploy window; auto-expire.

Testing alerts. amtool test routes; staging fire-drill (урок 4.4.2).

Grafana unified alerting. Can query Prometheus/Loki; multi-dimensional alerts; same routing discipline.

Как это выглядит на практике

On-call receives PagerDuty: ChurnHighErrorRate.

  1. Click runbook_url → triage 5.2.3.

  2. Grafana dashboard linked in annotation.

  3. Determines rollback needed → resolves alert after mitigation.

  4. Post-incident: threshold was too tight during low traffic — adjust for: 10m or add minimum RPS guard:

and sum(rate(istio_requests_total[5m])) > 1

Separate alert ChurnLatencyWarning → Slack only, p95 > 200ms for 15m — DS not woken at 3am for non-critical dev cluster (inhibit via cluster=prod label).

Weekly review: alerts that fired >3 times without action → tune or delete.

Что сделать после занятия

  • Напишите 2 alert rules для учебного сервиса (1 critical, 1 warning) с annotations.

  • Определите routing: что идёт in Slack vs pager.

  • Сформулируйте runbook link content (5 bullet triage steps).

Официальные материалы