Зачем это нужно
NVIDIA Triton Inference Server — high-performance server для нескольких framework'ов в одном процессе: ONNX, TensorRT, PyTorch, TensorFlow, Python backend. Dynamic batching, concurrent model instances, GPU sharing — то, что MLServer для tabular не даёт. Triton — стандарт для CV, NLP encoder, GPU-heavy inference в prod.
MLOps-инженер должен понимать, когда capstone LightGBM — MLServer, а image classification — Triton.
Основные идеи
Triton capabilities.
| Feature | Benefit |
|---|---|
| Multi-model | Many models one GPU |
| Dynamic batching | Merge requests → throughput ↑ |
| Model ensemble | Pre/post in Triton graph |
| Backend diversity | ONNX, TRT, PyTorch, Python |
| V2 protocol | Same as KServe/MLServer clients |
Repository layout (model repository):
models/ ├── churn_onnx/ │ ├── config.pbtxt │ └── 1/ │ └── model.onnx └── resnet50/ ├── config.pbtxt └── 1/ └── model.plan # TensorRT
Version folder 1/, 2/ — switch or A/B via load policy.
config.pbtxt (minimal ONNX):
name: "churn_onnx" platform: "onnxruntime_onnx" max_batch_size: 64 input [ { name: "input", data_type: TYPE_FP32, dims: [ 12 ] } ] output [ { name: "output", data_type: TYPE_FP32, dims: [ 1 ] } ] dynamic_batching { max_queue_delay_microseconds: 100 } instance_group [ { count: 2, kind: KIND_CPU } ]
Dynamic batching. Requests wait up to max_queue_delay to form batch → better GPU utilization; adds latency tail — tune for SLA.
Backends overview.
onnxruntime_onnx — portable, good default.
tensorrt_plan — fastest on NVIDIA GPU, less portable.
python — arbitrary code, GIL limits; use for pre/post only often.
pytorch_libtorch — TorchScript.
Triton vs MLServer.
| MLServer | Triton | |
|---|---|---|
| Tabular sklearn/LGBM | Excellent | Overkill |
| GPU DL models | Limited | Excellent |
| Dynamic batching | Basic | Advanced |
| Ops complexity | Lower | Higher |
KServe + Triton. ServingRuntime or prebuilt InferenceService with predictor.triton — platform pattern (7.5.2).
CPU-only Triton. Valid for ONNX sklearn-like models without GPU cluster; still get batching and multi-model.
Как это выглядит на практике
Export sklearn → ONNX for Triton:
`
training repo — one-time export
from skl2onnx import convert_sklearn
... produce model.onnx, upload to MinIO
`
Deploy on GPU node:
nodeSelector: nvidia.com/gpu.present: "true" resources: limits: nvidia.com/gpu: 1
Load test insight. Without dynamic batching: GPU util 20% at 100 RPS. With batching delay 5ms: util 70%, p95 +8ms — acceptable tradeoff.
Model warmup. First inference slow (CUDA kernels). Readiness probe should call infer or use warmup in config.
Что сделать после занятия
Объясните структуру model repository Triton (config + version folder).
Когда dynamic batching вреден для SLA?
Сравните Triton и MLServer для churn tabular (1 абзац).