7.5.2 · блок 7
KServe и runtimes
KServe и runtimes
Зачем это нужно
KServe абстрагирует Kubernetes plumbing, но качество inference определяет runtime: MLServer для tabular, Triton для GPU ONNX/TRT, vLLM для LLM. Platform team регистрирует ClusterServingRuntime; product team выбирает modelFormat и ресурсы.
Урок связывает 7.2–7.4 и 7.6 в единую карту «InferenceService → container».
Основные идеи
Mapping modelFormat → runtime.
| modelFormat | Typical image | Use case |
|-------------|---------------|----------|
| sklearn | MLServer | Tabular ML |
| lightgbm / xgboost | MLServer | Gradient boosting |
| onnx | MLServer or Triton | Portable DL/ML |
| triton | Triton server | GPU, multi-model |
| huggingface / custom | vLLM, TGI | LLM |
Exact mapping depends on cluster ServingRuntime definitions.
Triton via KServe:
spec:
predictor:
model:
modelFormat:
name: triton
storageUri: s3://ml-models/triton-repo/
runtime: kserve-tritonserver
resources:
limits:
nvidia.com/gpu: "1"
storageUri points to entire Triton model repository (multiple models possible).
MLServer sklearn (default path):
spec:
predictor:
model:
modelFormat:
name: sklearn
storageUri: s3://ml-models/churn/v44/
KServe downloads artifacts; MLServer loads via model-settings or convention.
Custom container (escape hatch):
spec:
predictor:
containers:
- name: kserve-container
image: registry.mdp.ru/ml/custom-vllm:0.6
args: ["--model", "meta-llama/Llama-3.2-3B"]
When pre-built format insufficient — full control, full responsibility.
Resource templates. Platform defaults in ServingRuntime; team overrides in InferenceService resources — requires quota approval for GPU.
Multi-model serving. One Triton InferenceService with repo containing embedder + classifier — client calls different model names on same host.
Autoscaling per runtime.
- MLServer CPU: HPA on CPU/concurrency.
- Triton GPU: KEDA, manual replicas; watch queue metrics.
- vLLM: GPU memory bound; often one model per GPU.
Security & tenancy. InferenceService in namespace ml-team-a; NetworkPolicy; OAuth at gateway; no cluster-wide admin for data scientists.
Как это выглядит на практике
MDP platform catalog (example):
| ClusterServingRuntime | Teams |
|-----------------------|-------|
| mlserver-sklearn-1.6 | Retail ML tabular |
| triton-24.05-gpu | CV, NLP encoders |
| vllm-0.6-llama | GenAI pilots |
Onboarding doc: «Pick runtime from catalog → copy InferenceService template → set storageUri».
Capstone path:
1. Week 1: MLServer sklearn InferenceService (CPU).
2. Optional stretch: export ONNX → Triton if GPU available.
3. Week 3: canary v45, observability dashboards.
Failure: format/runtime mismatch.
ModelLoadFailed: unsupported format pytorch for runtime mlserver-sklearn
Fix: change modelFormat or register correct ServingRuntime.
Что сделать после занятия
- [ ] Выберите runtime для capstone и заполните InferenceService spec.
- [ ] Объясните, зачем platform team централизует ServingRuntime versions.
- [ ] Сравните storageUri для MLServer (single model) vs Triton (repo).