MLOps Pathmlopspath.ru

07 · Serving-платформа7.1–7.7 · Runtime и оркестраторы7.6.3эксперт

Платформенная склейка LLM

Зачем это нужно

Один Pod с vLLM — demo. Платформа для GenAI в enterprise — gateway, auth, quotas, model registry, observability, cost allocation, guardrails. MLOps-инженер «склеивает» LLM runtime с компонентами модулей 3, 4, 5: Istio, Argo, ClearML/MMS, Prometheus, MinIO.

Урок — архитектурная карта для capstone stretch goals и стажировки на ML platform team.

Основные идеи

Reference architecture (layers).

┌─────────────────────────────────────────┐ │ API Gateway (Istio / OAuth2 / rate limit) │ ├─────────────────────────────────────────┤ │ Routing (model A vs B, A/B, fallback) │ ├─────────────────────────────────────────┤ │ KServe InferenceService (vLLM / Triton) │ ├─────────────────────────────────────────┤ │ Model artifacts (MinIO, HuggingFace mirror)│ ├─────────────────────────────────────────┤ │ GPU node pool + DCGM metrics │ ├─────────────────────────────────────────┤ │ Logs / traces / audit (Loki, Tempo) │ └─────────────────────────────────────────┘

Model lifecycle for LLM.

  1. Evaluate — ClearML log perplexity/benchmarks (offline).

  2. Register — MMS entry: model id, license, max context, approved use cases.

  3. Package — weights in MinIO, container image pin vLLM version.

  4. Deploy — InferenceService MR → Argo CD.

  5. Monitor — TTFT, tokens/sec, GPU util, error rate, policy violations.

  6. Retire — deprecate endpoint, archive weights per compliance.

Gateway concerns.

Concern Implementation
Authentication OAuth2/JWT at Istio
Authorization RBAC which teams call which model
Rate limit requests/min, tokens/day per API key
Request size max body, max image
PII scan/block patterns upstream

Prompt/response logging. Store metadata (latency, token count, model version); content logging opt-in with retention policy — legal review.

Fallback chain. Primary Llama → smaller backup model → static error message; circuit breaker when GPU pool exhausted.

Cost visibility. Label InferenceService by team, cost-center; Grafana dashboard GPU-hours × price.

RAG (Retrieval-Augmented Generation) — pointer. Platform pieces: vector DB, embedding service (Triton/vLLM embed model), orchestrator app. Full RAG — отдельный курс; здесь — embedding endpoint as sibling InferenceService.

GitOps everything. Model version bump = MR changing storageUri or args; no kubectl edit on prod vLLM.

Alignment with Google Rules of ML. Rule #28 launch and iterate; Rule #39 Monitor production; Rule #52 Give ML systems easy-to-use APIs — OpenAI-compatible gateway.

Как это выглядит на практике

Team onboarding checklist:

  • Namespace ml-genai-team-x quota 2×A10

  • ServiceAccount + MinIO read model bucket

  • InferenceService template from platform repo

  • API key in Vault → External Secret

  • Dashboard folder cloned from template

  • MMS registration approved by governance

Incident: all LLM requests 429.

  1. Check gateway rate limit config.

  2. Check vLLM queue depth / GPU OOM restarts.

  3. Scale replicas if capacity; else enable fallback model.

Capstone (optional GenAI track). Deploy small instruct model behind course gateway; integrate with observability from module 5; document in architecture C4 container diagram.

Что сделать после занятия

  • Нарисуйте 5-layer diagram для LLM platform в вашем стиле.

  • Выпишите 4 поля записи в MMS для LLM model card.

  • Опишите fallback strategy в 3 предложениях.

Официальные материалы