7.6.3 · блок 7

Платформенная склейка LLM

Зачем это нужно

Один Pod с vLLM — demo. Платформа для GenAI в enterprise — gateway, auth, quotas, model registry, observability, cost allocation, guardrails. MLOps-инженер «склеивает» LLM runtime с компонентами модулей 3, 4, 5: Istio, Argo, ClearML/MMS, Prometheus, MinIO.

Урок — архитектурная карта для capstone stretch goals и стажировки на ML platform team.

Основные идеи

Reference architecture (layers).


┌─────────────────────────────────────────┐
│  API Gateway (Istio / OAuth2 / rate limit) │
├─────────────────────────────────────────┤
│  Routing (model A vs B, A/B, fallback)     │
├─────────────────────────────────────────┤
│  KServe InferenceService (vLLM / Triton)   │
├─────────────────────────────────────────┤
│  Model artifacts (MinIO, HuggingFace mirror)│
├─────────────────────────────────────────┤
│  GPU node pool + DCGM metrics              │
├─────────────────────────────────────────┤
│  Logs / traces / audit (Loki, Tempo)       │
└─────────────────────────────────────────┘

Model lifecycle for LLM.

1. Evaluate — ClearML log perplexity/benchmarks (offline).

2. Register — MMS entry: model id, license, max context, approved use cases.

3. Package — weights in MinIO, container image pin vLLM version.

4. Deploy — InferenceService MR → Argo CD.

5. Monitor — TTFT, tokens/sec, GPU util, error rate, policy violations.

6. Retire — deprecate endpoint, archive weights per compliance.

Gateway concerns.

| Concern | Implementation |

|---------|----------------|

| Authentication | OAuth2/JWT at Istio |

| Authorization | RBAC which teams call which model |

| Rate limit | requests/min, tokens/day per API key |

| Request size | max body, max image |

| PII | scan/block patterns upstream |

Prompt/response logging. Store metadata (latency, token count, model version); content logging opt-in with retention policy — legal review.

Fallback chain. Primary Llama → smaller backup model → static error message; circuit breaker when GPU pool exhausted.

Cost visibility. Label InferenceService by team, cost-center; Grafana dashboard GPU-hours × price.

RAG (Retrieval-Augmented Generation) — pointer. Platform pieces: vector DB, embedding service (Triton/vLLM embed model), orchestrator app. Full RAG — отдельный курс; здесь — embedding endpoint as sibling InferenceService.

GitOps everything. Model version bump = MR changing storageUri or args; no kubectl edit on prod vLLM.

Alignment with Google Rules of ML. Rule #28 launch and iterate; Rule #39 Monitor production; Rule #52 Give ML systems easy-to-use APIs — OpenAI-compatible gateway.

Как это выглядит на практике

Team onboarding checklist:

[ ] Namespace ml-genai-team-x quota 2×A10
[ ] ServiceAccount + MinIO read model bucket
[ ] InferenceService template from platform repo
[ ] API key in Vault → External Secret
[ ] Dashboard folder cloned from template
[ ] MMS registration approved by governance

Incident: all LLM requests 429.

1. Check gateway rate limit config.

2. Check vLLM queue depth / GPU OOM restarts.

3. Scale replicas if capacity; else enable fallback model.

Capstone (optional GenAI track). Deploy small instruct model behind course gateway; integrate with observability from module 5; document in architecture C4 container diagram.

Что сделать после занятия

[ ] Нарисуйте 5-layer diagram для LLM platform в вашем стиле.
[ ] Выпишите 4 поля записи в MMS для LLM model card.
[ ] Опишите fallback strategy в 3 предложениях.

Официальные материалы

Открыть интерактивную версию