Зачем это нужно
KServe абстрагирует Kubernetes plumbing, но качество inference определяет runtime: MLServer для tabular, Triton для GPU ONNX/TRT, vLLM для LLM. Platform team регистрирует ClusterServingRuntime; product team выбирает modelFormat и ресурсы.
Урок связывает 7.2–7.4 и 7.6 в единую карту «InferenceService → container».
Основные идеи
Mapping modelFormat → runtime.
| modelFormat | Typical image | Use case |
|---|---|---|
| sklearn | MLServer | Tabular ML |
| lightgbm / xgboost | MLServer | Gradient boosting |
| onnx | MLServer or Triton | Portable DL/ML |
| triton | Triton server | GPU, multi-model |
| huggingface / custom | vLLM, TGI | LLM |
Exact mapping depends on cluster ServingRuntime definitions.
Triton via KServe:
spec: predictor: model: modelFormat: name: triton storageUri: s3://ml-models/triton-repo/ runtime: kserve-tritonserver resources: limits: nvidia.com/gpu: "1"
storageUri points to ** entire Triton model repository** (multiple models possible).
MLServer sklearn (default path):
spec: predictor: model: modelFormat: name: sklearn storageUri: s3://ml-models/churn/v44/
KServe downloads artifacts; MLServer loads via model-settings or convention.
Custom container (escape hatch):
spec: predictor: containers: - name: kserve-container image: registry.mdp.ru/ml/custom-vllm:0.6 args: ["--model", "meta-llama/Llama-3.2-3B"]
When pre-built format insufficient — full control, full responsibility.
Resource templates. Platform defaults in ServingRuntime; team overrides in InferenceService resources — requires quota approval for GPU.
Multi-model serving. One Triton InferenceService with repo containing embedder + classifier — client calls different model names on same host.
Autoscaling per runtime.
MLServer CPU: HPA on CPU/concurrency.
Triton GPU: KEDA, manual replicas; watch queue metrics.
vLLM: GPU memory bound; often one model per GPU.
Security & tenancy. InferenceService in namespace ml-team-a; NetworkPolicy; OAuth at gateway; no cluster-wide admin for data scientists.
Как это выглядит на практике
MDP platform catalog (example):
| ClusterServingRuntime | Teams |
|---|---|
| mlserver-sklearn-1.6 | Retail ML tabular |
| triton-24.05-gpu | CV, NLP encoders |
| vllm-0.6-llama | GenAI pilots |
Onboarding doc: «Pick runtime from catalog → copy InferenceService template → set storageUri».
Capstone path:
Week 1: MLServer sklearn InferenceService (CPU).
Optional stretch: export ONNX → Triton if GPU available.
Week 3: canary v45, observability dashboards.
Failure: format/runtime mismatch.
ModelLoadFailed: unsupported format pytorch for runtime mlserver-sklearn
Fix: change modelFormat or register correct ServingRuntime.
Что сделать после занятия
Выберите runtime для capstone и заполните InferenceService spec.
Объясните, зачем platform team централизует ServingRuntime versions.
Сравните storageUri для MLServer (single model) vs Triton (repo).