5.3.2 · блок 5

Alerting: от метрики до действия

Зачем это нужно

Dashboard без алертов — музей. On-call просыпается, когда пользователи уже пострадали, если нет проактивных правил. Плохие алерты — «alert fatigue»: команда игнорирует всё. Хорошие алерты — actionable, привязаны к симптомам и runbook.

Для ML inference алертируем infra (RED, GPU) и позже quality (drift) — разные severity и каналы.

Основные идеи

Alert pipeline.


Prometheus/VM rule → Alertmanager → route → PagerDuty/Slack/email
Grafana managed alerts → contact point (alternative path)

Prometheus alerting rule example.


groups:
  - name: churn-serving
    rules:
      - alert: ChurnHighErrorRate
        expr: |
          sum(rate(istio_requests_total{destination_service="churn-serving",response_code=~"5.."}[5m]))
          /
          sum(rate(istio_requests_total{destination_service="churn-serving"}[5m]))
          > 0.05
        for: 5m
        labels:
          severity: critical
          service: churn-serving
        annotations:
          summary: "Churn error rate above 5%"
          runbook_url: "https://wiki/runbooks/churn-errors"

for duration. Условие должно держаться N минут — фильтр flapping.

Severity levels.

| Level | Meaning | Example |

|-------|---------|---------|

| critical | User-facing outage | error rate > 5%, all pods down |

| warning | Degraded, time to act | p95 latency 2× SLO, GPU mem 90% |

| info | FYI | deploy completed |

Alertmanager routing.

severity=critical → PagerDuty on-call.
severity=warning → Slack #ml-ops.
team=ds → drift alerts (5.3.4).

Actionable annotations. summary, description, runbook_url, dashboard_url. «Something is wrong» — плохо.

Symptom-based vs cause-based.

Symptom: high error rate, SLO burn — page on-call.
Cause: «disk 80%» — ticket unless causes outage soon.

ML-specific alerts.

Inference RED breaches.
All replicas not Ready after deploy (Argo + kube alerts).
GPU XID errors.
Prediction distribution shift (batch job alert — not always real-time page).

Silences and maintenance. Alertmanager silence during planned deploy window; auto-expire.

Testing alerts. amtool test routes; staging fire-drill (урок 4.4.2).

Grafana unified alerting. Can query Prometheus/Loki; multi-dimensional alerts; same routing discipline.

Как это выглядит на практике

On-call receives PagerDuty: ChurnHighErrorRate.

1. Click runbook_url → triage 5.2.3.

2. Grafana dashboard linked in annotation.

3. Determines rollback needed → resolves alert after mitigation.

4. Post-incident: threshold was too tight during low traffic — adjust for: 10m or add minimum RPS guard:


and sum(rate(istio_requests_total[5m])) > 1

Separate alert ChurnLatencyWarning → Slack only, p95 > 200ms for 15m — DS not woken at 3am for non-critical dev cluster (inhibit via cluster=prod label).

Weekly review: alerts that fired >3 times without action → tune or delete.

Что сделать после занятия

[ ] Напишите 2 alert rules для учебного сервиса (1 critical, 1 warning) с annotations.
[ ] Определите routing: что идёт in Slack vs pager.
[ ] Сформулируйте runbook link content (5 bullet triage steps).

Официальные материалы

Открыть интерактивную версию