7.4.2 · блок 7

Эксплуатация Triton

Зачем это нужно

Поднять Triton demo — полдела. Эксплуатация — GPU capacity, model reload без downtime, metrics, failure modes (CUDA OOM, wrong input shape). Этот урок для MLE/SRE: как Triton живёт в K8s неделями, не минутами demo.

Основные идеи

Lifecycle operations.

| Operation | Mechanism |

|-----------|-----------|

| Load model | Add to repo + `POST /v2/repository/models/{name}/load` |

| Unload | Free GPU memory |

| Hot swap version | New folder `2/`, load, switch traffic, unload `1/` |

| Full restart | Pod restart — brief outage unless multi-replica |

Health endpoints.

/v2/health/live — server up.
/v2/health/ready — models loaded and ready.
/v2/models/{name}/ready — per-model.

Configure K8s probes on ready after model load completes.

Metrics (Prometheus).

Key metrics (names may vary by version):

nv_inference_request_success / fail
nv_inference_queue_duration_us
nv_inference_compute_infer_duration_us
nv_gpu_utilization (with DCGM integration)
nv_inference_count

Dashboard: RED on API + USE on GPU (module 5.1).

GPU memory management.

Each model instance allocates weights + workspace.
Too many instance_group → CUDA OOM → Pod crash.
Model analyzer tool — recommends instance count and batch size for target SLA.

Autoscaling.

HPA on CPU insufficient for GPU workloads.
Options: KEDA on queue depth / custom metric RPS; manual replica count; time-based scale for predictable peaks.
KServe scale-to-zero with GPU — cold start painful; minReplicas ≥ 1 typical.

Input validation errors. Wrong shape → 400; log request id, not full payload if PII. Alert spike in 4xx.

Multi-tenant Triton. One Triton per team vs shared platform Triton — isolation vs utilization tradeoff. Shared: quotas, separate model namespaces in repo, NetworkPolicy.

Security.

No public Ingress without auth (OAuth2 proxy, Istio JWT).
Read-only model volume; signed artifacts from MinIO.

Backup. Model repo in object storage; config.pbtxt in Git. Pod ephemeral.

Как это выглядит на практике

Incident: p95 latency spike

1. Grafana: queue duration ↑, compute flat → input backlog, need more instances or batch tuning.

2. Grafana: GPU util 99%, compute ↑ → capacity — add replica or TensorRT optimize.

3. Error rate ↑ OOM → reduce instance_group count or model size.

Runbook excerpt:


Symptom: Triton Pod OOMKilled
1. kubectl describe pod → Last State OOM
2. Check config.pbtxt instance_group count
3. Reduce count or request larger GPU (A10 → A100)
4. Rollout via GitOps
5. Verify nv_gpu_memory_used_bytes stable 15 min

Rolling update with KServe.

New InferenceService revision with updated storageUri.
Canary traffic 10% → compare metrics → promote.

Perf test before prod.

perf_analyzer against Triton in staging.
Document max RPS at p95 target for capacity planning.

Что сделать после занятия

[ ] Выпишите 3 Prometheus метрики Triton и что они означают.
[ ] Опишите шаги hot swap model version без полного outage (2 replicas).
[ ] Составьте симптом→диагностика для CUDA OOM.

Официальные материалы

Открыть интерактивную версию