7.4.1 · блок 7

NVIDIA Triton как универсальный server

Зачем это нужно

NVIDIA Triton Inference Server — high-performance server для нескольких framework'ов в одном процессе: ONNX, TensorRT, PyTorch, TensorFlow, Python backend. Dynamic batching, concurrent model instances, GPU sharing — то, что MLServer для tabular не даёт. Triton — стандарт для CV, NLP encoder, GPU-heavy inference в prod.

MLOps-инженер должен понимать, когда capstone LightGBM — MLServer, а image classification — Triton.

Основные идеи

Triton capabilities.

| Feature | Benefit |

|---------|---------|

| **Multi-model** | Many models one GPU |

| **Dynamic batching** | Merge requests → throughput ↑ |

| **Model ensemble** | Pre/post in Triton graph |

| **Backend diversity** | ONNX, TRT, PyTorch, Python |

| **V2 protocol** | Same as KServe/MLServer clients |

Repository layout (model repository):


models/
├── churn_onnx/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx
└── resnet50/
    ├── config.pbtxt
    └── 1/
        └── model.plan   # TensorRT

Version folder 1/, 2/ — switch or A/B via load policy.

config.pbtxt (minimal ONNX):


name: "churn_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 64
input [
  { name: "input", data_type: TYPE_FP32, dims: [ 12 ] }
]
output [
  { name: "output", data_type: TYPE_FP32, dims: [ 1 ] }
]
dynamic_batching {
  max_queue_delay_microseconds: 100
}
instance_group [
  { count: 2, kind: KIND_CPU }
]

Dynamic batching. Requests wait up to max_queue_delay to form batch → better GPU utilization; adds latency tail — tune for SLA.

Backends overview.

onnxruntime_onnx — portable, good default.
tensorrt_plan — fastest on NVIDIA GPU, less portable.
python — arbitrary code, GIL limits; use for pre/post only often.
pytorch_libtorch — TorchScript.

Triton vs MLServer.

| | MLServer | Triton |

|---|----------|--------|

| Tabular sklearn/LGBM | Excellent | Overkill |

| GPU DL models | Limited | Excellent |

| Dynamic batching | Basic | Advanced |

| Ops complexity | Lower | Higher |

KServe + Triton. ServingRuntime or prebuilt InferenceService with predictor.triton — platform pattern (7.5.2).

CPU-only Triton. Valid for ONNX sklearn-like models without GPU cluster; still get batching and multi-model.

Как это выглядит на практике

Export sklearn → ONNX for Triton:


# training repo — one-time export
from skl2onnx import convert_sklearn
# ... produce model.onnx, upload to MinIO

Deploy on GPU node:


nodeSelector:
  nvidia.com/gpu.present: "true"
resources:
  limits:
    nvidia.com/gpu: 1

Load test insight. Without dynamic batching: GPU util 20% at 100 RPS. With batching delay 5ms: util 70%, p95 +8ms — acceptable tradeoff.

Model warmup. First inference slow (CUDA kernels). Readiness probe should call infer or use warmup in config.

Что сделать после занятия

[ ] Объясните структуру model repository Triton (config + version folder).
[ ] Когда dynamic batching вреден для SLA?
[ ] Сравните Triton и MLServer для churn tabular (1 абзац).

Официальные материалы

Открыть интерактивную версию