MLOps Pathmlopspath.ru

07 · Serving-платформа7.1–7.7 · Runtime и оркестраторы7.4.2эксперт

Эксплуатация Triton

Зачем это нужно

Поднять Triton demo — полдела. Эксплуатация — GPU capacity, model reload без downtime, metrics, failure modes (CUDA OOM, wrong input shape). Этот урок для MLE/SRE: как Triton живёт в K8s неделями, не минутами demo.

Основные идеи

Lifecycle operations.

Operation Mechanism
Load model Add to repo + POST /v2/repository/models/{name}/load
Unload Free GPU memory
Hot swap version New folder 2/, load, switch traffic, unload 1/
Full restart Pod restart — brief outage unless multi-replica

Health endpoints.

  • /v2/health/live — server up.

  • /v2/health/ready — models loaded and ready.

  • /v2/models/{name}/ready — per-model.

Configure K8s probes on ready after model load completes.

Metrics (Prometheus).

Key metrics (names may vary by version):

  • nv_inference_request_success / fail

  • nv_inference_queue_duration_us

  • nv_inference_compute_infer_duration_us

  • nv_gpu_utilization (with DCGM integration)

  • nv_inference_count

Dashboard: RED on API + USE on GPU (module 5.1).

GPU memory management.

  • Each model instance allocates weights + workspace.

  • Too many instance_group → CUDA OOM → Pod crash.

  • Model analyzer tool — recommends instance count and batch size for target SLA.

Autoscaling.

  • HPA on CPU insufficient for GPU workloads.

  • Options: KEDA on queue depth / custom metric RPS; manual replica count; time-based scale for predictable peaks.

  • KServe scale-to-zero with GPU — cold start painful; minReplicas ≥ 1 typical.

Input validation errors. Wrong shape → 400; log request id, not full payload if PII. Alert spike in 4xx.

Multi-tenant Triton. One Triton per team vs shared platform Triton — isolation vs utilization tradeoff. Shared: quotas, separate model namespaces in repo, NetworkPolicy.

Security.

  • No public Ingress without auth (OAuth2 proxy, Istio JWT).

  • Read-only model volume; signed artifacts from MinIO.

Backup. Model repo in object storage; config.pbtxt in Git. Pod ephemeral.

Как это выглядит на практике

Incident: p95 latency spike

  1. Grafana: queue duration ↑, compute flat → input backlog, need more instances or batch tuning.

  2. Grafana: GPU util 99%, compute ↑ → capacity — add replica or TensorRT optimize.

  3. Error rate ↑ OOM → reduce instance_group count or model size.

Runbook excerpt:

` Symptom: Triton Pod OOMKilled

  1. kubectl describe pod → Last State OOM
  2. Check config.pbtxt instance_group count
  3. Reduce count or request larger GPU (A10 → A100)
  4. Rollout via GitOps
  5. Verify nv_gpu_memory_used_bytes stable 15 min `

Rolling update with KServe.

  • New InferenceService revision with updated storageUri.

  • Canary traffic 10% → compare metrics → promote.

Perf test before prod.

  • perf_analyzer against Triton in staging.

  • Document max RPS at p95 target for capacity planning.

Что сделать после занятия

  • Выпишите 3 Prometheus метрики Triton и что они означают.

  • Опишите шаги hot swap model version без полного outage (2 replicas).

  • Составьте симптом→диагностика для CUDA OOM.

Официальные материалы