7.3.2 · блок 7

Seldon и разные runtimes

Зачем это нужно

Этот урок разбирает legacy Seldon Core 1: runtime внутри graph выполняет inference. Один graph может комбинировать sklearn preprocessor (MLServer), ONNX model (ONNX Runtime server), TensorFlow SavedModel. SeldonDeployment и implementation ниже нужны для миграции/поддержки Core 1; для нового production выбирайте Seldon Core 2 или KServe как golden path.

Урок дополняет 7.3.1 практикой multi-runtime и migration path к KServe.

Основные идеи

Pre-packaged Seldon Core 1 servers (historical names).

| implementation | Backend | Typical model |

|----------------|---------|---------------|

| SKLEARN_SERVER | MLServer/sklearn | joblib |

| XGBOOST_SERVER | MLServer | xgboost |

| MLFLOW_SERVER | MLflow flavor | pyfunc |

| TENSORFLOW_SERVER | TF Serving | SavedModel |

| ONNX_SERVER | ONNX Runtime | .onnx |

| TRITON_SERVER | NVIDIA Triton | multiple formats |

Exact names evolve; check Seldon version docs. Trend: MLServer and Triton as primary.

modelUri schemes.

gs://, s3://, hdfs://, http:// — initContainer download to local volume.
Secret credentials via secretRef in SeldonDeployment.

Custom runtime. implementation: CUSTOM + your Docker image implementing Seldon API or V2 protocol. Use when pre-packaged insufficient.

Multi-container predictor. componentSpecs — per-node resource limits, env, affinity (GPU node for Triton child, CPU for transformer).

Example: ONNX classifier after Python transformer


predictors:
  - graph:
      name: preprocess
      type: TRANSFORMER
      implementation: CUSTOM
      modelUri: "s3://ml-models/churn/preprocess/"
      envSecretRefName: minio-credentials
      children:
        - name: onnx-clf
          type: MODEL
          implementation: ONNX_SERVER
          modelUri: "s3://ml-models/churn/v44/model.onnx"
    componentSpecs:
      - spec:
          containers:
            - name: onnx-clf
              resources:
                limits:
                  nvidia.com/gpu: "0"  # CPU ONNX OK

Triton under Seldon. Triton excels GPU batching, multi-model on one GPU. Seldon graph node → Triton container; ops team manages CUDA drivers (module 5.1.3 GPU metrics).

MLServer under Seldon vs KServe MLServer. Functionally similar container; KServe adds InferenceService abstraction (canary, scale-to-zero, storage initializer). Greenfield MDP capstone → prefer KServe (7.5).

Version skew. Preprocessor v2 + model v44 trained on v1 features → silent quality drop. Version together in one Git MR.

Inference graph testing.

Unit tests on transformer code (repo).
Contract test: golden JSON in → expected tensor shape to model node.
Integration: SeldonDeployment in dev namespace + smoke curl.

Как это выглядит на практике

Migration story: legacy Seldon Core 1 → KServe

1. Phase 1: export all models to ONNX + MLServer settings.

2. Phase 2: parallel deploy KServe InferenceService, mirror traffic Istio mirror.

3. Phase 3: cutover, deprecate SeldonDeployment.

Operational checklist per runtime.

| Runtime | Watch |

|---------|-------|

| MLServer | Model load time, worker count |

| Triton | GPU util, queue time, model instances |

| TF Serving | Graph def version, batching config |

| Custom | Your metrics, log stack traces |

Resource planning. Two-node graph = two containers possibly on same Pod (Seldon executor model) or separate — depends on Seldon architecture version. Read resource sum for HPA.

Что сделать после занятия

[ ] Для capstone выберите implementation (SKLEARN vs ONNX) и обоснуйте.
[ ] Опишите риск version skew между transformer и model.
[ ] Найдите в Seldon docs один пример modelUri с S3/MinIO.

Официальные материалы

Открыть интерактивную версию