8.1.3 · блок 8
Capstone: неделя 3 — prod и защита
Capstone: неделя 3 — prod и защита
Зачем это нужно
Третья неделя — production readiness и защита проекта. Prod здесь — учебный ml-capstone-prod с реальными практиками: canary, observability, runbook, security, MMS approval. Защита — не только «F1=0.82», а демонстрация MLOps-зрелости: вы бы доверили этому сервису on-call?
Основные идеи
Цель недели 3. Prod deploy с canary или blue-green; dashboards RED + one ML quality signal; runbook; успешная защита 15–20 мин.
Deliverables checklist.
| # | Артефакт | Модули |
|---|----------|--------|
| 1 | Prod InferenceService + GitOps | 7.5, 3.7 |
| 2 | Canary or version rollout v1→v2 | 7.5.1 |
| 3 | Grafana dashboard RED (+ GPU if used) | 5.1 |
| 4 | Log query example (Loki) | 5.2 |
| 5 | One alert rule (latency or errors) | 5.3 |
| 6 | SLI/SLO draft (even simple) | 5.3.3 |
| 7 | Runbook 1 page | 5.2.3 |
| 8 | MMS prod registration + approval record | 4.3 |
| 9 | Security: NetworkPolicy, no secrets leak | 3.3.6 |
| 10 | Final C4 + ADR summary | 2.2 |
Prod promotion flow.
1. v2 trained and registered (week 2 pipeline).
2. MR values-prod.yaml: canary 10% traffic or parallel revision.
3. MMS approval ticket closed.
4. Argo sync prod Application.
5. 24h monitor (in course — 2–4 hours simulated).
6. Promote to 100% or rollback Git revert.
Observability minimum viable.
- RED: RPS, 5xx rate, p95 latency on
/infer. - ML quality proxy: null feature rate, prediction distribution drift (5.3.4) — один график.
- Logs: trace one request id through gateway → pod.
SLO example (tabular churn).
- Availability: 99.5% monthly ( учебный ).
- Latency: p95 < 300 ms excluding client network.
- Error budget: if 5xx > 0.5% for 1h → freeze releases.
Runbook sections.
1. Symptoms (latency up, errors up, model load fail).
2. Checks (Grafana links, kubectl describe isvc).
3. Mitigations (rollback Git SHA, scale replicas).
4. Escalation (platform vs ML owner).
Security checklist.
- [ ] Ingress TLS + auth (JWT or API key).
- [ ] Secrets from External Secrets Operator.
- [ ] NetworkPolicy: ingress only from gateway namespace.
- [ ] Container image scanned in CI (mention policy).
- [ ] PII not in logs.
Defense presentation structure (15 min).
1. Problem & SLA (2 min) — business case, online vs batch.
2. Architecture (3 min) — C4 container, data path block 6.
3. Live demo (4 min) — infer + dashboard + rollback story.
4. MLOps process (3 min) — CI/CD, GitOps, MMS, quality gate.
5. Ops & risks (2 min) — SLO, runbook, known limitations.
6. Q&A (5 min).
Grading rubric (orientir).
| Критерий | Вес |
|----------|-----|
| Working prod infer | 25% |
| Pipeline reproducibility | 20% |
| Observability | 15% |
| GitOps / no manual prod | 15% |
| Documentation & ADR | 10% |
| Defense clarity | 15% |
F1 alone — не более 10% implicit inside reproducibility.
Rollback demo (сильный ход). На защите показать revert MR → Argo → версия v1 снова 100% — доказывает зрелость.
Как это выглядит на практике
Понедельник: prod manifests, MMS approval, dashboard import.
Вторник: canary v2; tune alert.
Среда: chaos lite — kill one pod, show HPA/recovery; or inject bad payload → 4xx metrics.
Четверг: rehearsal defense; fix README.
Пятница: защита.
If things go wrong.
| Situation | Plan B |
|-----------|--------|
| Prod deploy broken | Demo dev + explain prod diff honestly |
| v2 worse than v1 | Rollback is feature — show it |
| No GPU for LLM stretch | Tabular path fully valid |
Post-capstone. Export repo as portfolio; link architecture in CV; optional blog post «what I learned».
Что сделать после занятия
- [ ] Complete prod checklist 10/10 or document waived items with ADR.
- [ ] Rehearse demo script timed 12 minutes.
- [ ] Peer review runbook: can teammate follow without you?