Зачем это нужно
На второй неделе каркас наполняется содержанием: данные, features, обучение, registry, deploy реальной модели. Это склеивание блоков 4 (MLOps lifecycle), 6 (Data), 7 (Serving). Цель — воспроизводимый pipeline «dataset → model artifact → dev inference» с experiment tracking.
Основные идеи
Цель недели 2. Обученная модель v1 в registry; InferenceService dev отдаёт predictions на golden test set; ClearML/MMS фиксирует эксперимент и dataset version.
Deliverables checklist.
| # | Артефакт | Модули |
|---|---|---|
| 1 | Data snapshot + quality checks | 6.1 |
| 2 | Feature pipeline (Spark or Python) | 6.3 |
| 3 | Optional Feast feature views | 6.4 |
| 4 | Train script + ClearML logging | 4.3 |
| 5 | Model registration MMS/dev registry | 4.3 |
| 6 | Export model for MLServer | 7.2 |
| 7 | Update storageUri / image tag |
7.5 |
| 8 | Integration test: infer vs expected | 4.4 |
| 9 | Argo Workflow OR Jenkins stage train | 3.7, 3.2 |
| 10 | Data/model card in repo | 4.1 |
Pipeline DAG (target).
ingest → validate → features → train → evaluate gate → register → export → deploy dev
Не обязательно всё automated week 2 — manual trigger OK, но скрипты в Git.
Data quality gate (6.1.2). Train не стартует если null_rate или schema failed — покажите один реальный check в demo.
Experiment tracking minimum.
Log: hyperparams, metrics, dataset path/hash, git commit.
Tag:
capstone-v1.Artifact: model file + confusion matrix plot.
Feature scope discipline. 8–15 фич достаточно; лучше working pipeline, чем 100 фич без serve path.
Train/serve alignment. Если Feast — get_historical_features in train, get_online_features in transformer (7.5.2). If no Feast — shared Python package imported in train and serving container same version.
Evaluation gate. Example: F1 ≥ baseline 0.65 on hold-out — иначе no register. Business metric optional.
Deploy real model to dev.
spec: predictor: model: modelFormat: name: sklearn storageUri: s3://ml-capstone/models/churn/v1/
Run smoke from Jenkins after deploy.
Kafka/Spark optional. Если в ADR (6.4.3) выбран streaming — минимум: один topic + consumer lag dashboard or one Spark batch job output Parquet. Не блокируйте v1 на infra.
Definition of Done недели 2.
ClearML/MMS shows experiment linked to dataset v1
Dev inference returns non-stub scores for fixture JSON
Integration test in CI (may run on schedule if GPU slow)
Model card: limitations, metrics, features list
Peer review MR deploy + train code
Как это выглядит на практике
Понедельник: finalize dataset; run quality checks.
Вторник–среда: train v1; iterate if metrics poor.
Четверг: register + export; update KServe dev.
Пятница: demo — live infer + ClearML screen + data lineage story 3 min.
Typical failures.
| Problem | Fix |
|---|---|
| Feature mismatch infer | Align schema OpenAPI vs model n_features |
| Model load fail | Check storageUri credentials |
| Metrics good in notebook, bad in pipeline | Different split/leakage — fix PIT join |
| CI train timeout | Smaller sample job for CI; full train manual |
Documentation update. C4 Container diagram: add Spark, MinIO, ClearML, KServe edges.
Что сделать после занятия
Run full train → register → deploy once without manual kubectl.
Attach dataset hash to MMS/ClearML run.
Write model card section «Known limitations».