5.3.3 · блок 5
SLI, SLO и error budget
SLI, SLO и error budget
Зачем это нужно
«99.9% uptime» без определения — спор на ретро. SLI (indicator) измеряет аспект сервиса; SLO (objective) — целевое значение; error budget — сколько «плохого» поведения допустимо до freeze фич и focus на reliability.
Для ML inference SLO часто на latency + availability; quality SLO (accuracy proxy) — отдельно и сложнее. Этот урок — язык SRE для согласования с бизнесом и Platform.
Основные идеи
Definitions.
| Term | Meaning | Example |
|------|---------|---------|
| **SLI** | Metric + measurement window | % successful `/predict` requests < 500ms |
| **SLO** | Target for SLI | 99.5% of requests/month |
| **SLA** | Contract with penalties | 99.9% or refund — legal, rare internal |
| **Error budget** | 100% - SLO | 0.5% bad requests/month allowed |
Good SLI properties.
- User-centric (what customer feels).
- Measurable from telemetry (Istio + app).
- Stable definition over time.
Inference SLI examples.
1. Availability SLI: successful requests / total requests (exclude 4xx client fault if policy says so).
2. Latency SLI: requests faster than 200ms / total.
3. Composite: both must pass for «good» request.
PromQL sketch (latency SLI ratio):
sum(rate(istio_request_duration_milliseconds_bucket{le="200",destination_service="churn-serving"}[5m]))
/
sum(rate(istio_request_duration_milliseconds_count{destination_service="churn-serving"}[5m]))
Error budget policy.
- Budget exhausted → freeze deploys, focus reliability sprint.
- Budget healthy → allow aggressive canary, tech debt paydown lower priority.
Два способа считать бюджет. Request-based SLI считает долю плохих запросов: при SLO 99.5% допустимы 0.5% bad events. Time-based SLI считает долю времени, когда сервис был недоступен: при том же SLO допустимы 0.5% времени окна. Это разные показатели и их нельзя переводить друг в друга без предположения о нагрузке.
Multi-window burn alerts (Google SRE). Fast burn (1h) and slow burn (6h) on budget consumption — page before total monthly exhaustion.
SLO vs alerting. Alert on budget burn, not only instantaneous spike — reduces false pages.
ML nuance. Model can meet latency SLO but wrong predictions — not in classical SLI; extend with quality SLO (5.3.4) or business KPI dashboards with separate process.
Choosing SLO targets. Too strict → no innovation; too loose → users churn. Start from historical p99 + business ask; review quarterly.
Documentation. SLO page in Confluence: SLI definition, query, target, error budget policy, owners.
Как это выглядит на практике
Churn scoring SLO (internal):
- SLI: proportion of
/predictwith status 2xx AND duration < 200ms. - SLO: 99.5% over 30 days.
- Error budget: 0.5% bad requests за 30 дней. Время «216 минут» относится только к отдельному time-based availability SLO (0.5% от 30 дней), а не напрямую к request-based SLI.
Grafana SLO dashboard:
- Remaining budget gauge.
- Burn rate panels 1h / 24h.
Scenario: team shipped 5 model releases this week; latency regressions burned 80% budget.
- Platform declares «deploy freeze» except hotfix.
- DS optimizes inference batching; MLE adds cache for features.
- Next month reset budget.
Meeting with PM: agree on either p95 < 200ms as percentile objective or 99.5% requests < 200ms as request-based objective; это разные SLI и их нельзя подменять друг другом. Business accepts 99.5 not 99.99 because cost of redundant GPU.
Distinction: monitoring shows graphs; SLO drives decisions.
Что сделать после занятия
- [ ] Определите 1 SLI и SLO для учебного API (availability или latency).
- [ ] Посчитите error budget при SLO 99.9% за 30 дней (rough minutes of downtime).
- [ ] Опишите policy: что делает команда при 90% budget spent.