MLOps Pathmlopspath.ru

07 · Serving-платформа7.1–7.7 · Runtime и оркестраторы7.3.2сложный

Seldon и разные runtimes

Зачем это нужно

Этот урок разбирает legacy Seldon Core 1: runtime внутри graph выполняет inference. Один graph может комбинировать sklearn preprocessor (MLServer), ONNX model (ONNX Runtime server), TensorFlow SavedModel. SeldonDeployment и implementation ниже нужны для миграции/поддержки Core 1; для нового production выбирайте Seldon Core 2 или KServe как golden path.

Урок дополняет 7.3.1 практикой multi-runtime и migration path к KServe.

Основные идеи

Pre-packaged Seldon Core 1 servers (historical names).

implementation Backend Typical model
SKLEARN_SERVER MLServer/sklearn joblib
XGBOOST_SERVER MLServer xgboost
MLFLOW_SERVER MLflow flavor pyfunc
TENSORFLOW_SERVER TF Serving SavedModel
ONNX_SERVER ONNX Runtime .onnx
TRITON_SERVER NVIDIA Triton multiple formats

Exact names evolve; check Seldon version docs. Trend: MLServer and Triton as primary.

modelUri schemes.

  • gs://, s3://, hdfs://, http:// — initContainer download to local volume.

  • Secret credentials via secretRef in SeldonDeployment.

Custom runtime. implementation: CUSTOM + your Docker image implementing Seldon API or V2 protocol. Use when pre-packaged insufficient.

Multi-container predictor. componentSpecs — per-node resource limits, env, affinity (GPU node for Triton child, CPU for transformer).

Example: ONNX classifier after Python transformer

` predictors:

  • graph: name: preprocess type: TRANSFORMER implementation: CUSTOM modelUri: "s3://ml-models/churn/preprocess/" envSecretRefName: minio-credentials children: - name: onnx-clf type: MODEL implementation: ONNX_SERVER modelUri: "s3://ml-models/churn/v44/model.onnx" componentSpecs:
    • spec: containers: - name: onnx-clf resources: limits: nvidia.com/gpu: "0" # CPU ONNX OK

`

Triton under Seldon. Triton excels GPU batching, multi-model on one GPU. Seldon graph node → Triton container; ops team manages CUDA drivers (module 5.1.3 GPU metrics).

MLServer under Seldon vs KServe MLServer. Functionally similar container; KServe adds InferenceService abstraction (canary, scale-to-zero, storage initializer). Greenfield MDP capstone → prefer KServe (7.5).

Version skew. Preprocessor v2 + model v44 trained on v1 features → silent quality drop. Version together in one Git MR.

Inference graph testing.

  • Unit tests on transformer code (repo).

  • Contract test: golden JSON in → expected tensor shape to model node.

  • Integration: SeldonDeployment in dev namespace + smoke curl.

Как это выглядит на практике

Migration story: legacy Seldon Core 1 → KServe

  1. Phase 1: export all models to ONNX + MLServer settings.

  2. Phase 2: parallel deploy KServe InferenceService, mirror traffic Istio mirror.

  3. Phase 3: cutover, deprecate SeldonDeployment.

Operational checklist per runtime.

Runtime Watch
MLServer Model load time, worker count
Triton GPU util, queue time, model instances
TF Serving Graph def version, batching config
Custom Your metrics, log stack traces

Resource planning. Two-node graph = two containers possibly on same Pod (Seldon executor model) or separate — depends on Seldon architecture version. Read resource sum for HPA.

Что сделать после занятия

  • Для capstone выберите implementation (SKLEARN vs ONNX) и обоснуйте.

  • Опишите риск version skew между transformer и model.

  • Найдите в Seldon docs один пример modelUri с S3/MinIO.

Официальные материалы