Зачем это нужно
Vision-Language Models (VLM) принимают изображение + текст («что на фото?», OCR, visual QA). Payload больше, preprocessing сложнее, latency выше, чем у tabular ML. MLOps-инженер должен понимать multimodal inference path: upload → storage → resize/tokenize → GPU forward → response.
Урок расширяет serving stack на CV+LLM без глубокого DL — фокус на операционных отличиях.
Основные идеи
Multimodal request anatomy.
Client → API Gateway ├── image (base64 or presigned URL) ├── text prompt └── optional parameters (max_tokens, temperature) → Preprocessor (resize, normalize, patch embed) → VLM model (GPU) → Text (and optionally structured JSON) response
Payload size problem. 5 MB image in JSON base64 — bad for Kafka/gateway limits. Pattern: presigned URL to object storage (MinIO); inference Pod downloads internally.
Runtimes for VLM.
| Stack | Notes |
|---|---|
| vLLM | Growing multimodal support (model-dependent) |
| Triton + Python/ONNX | Custom pre/post in backend |
| Dedicated (LMDeploy, etc.) | Alternative engines |
Check model card for supported server.
Preprocessing consistency. Train used 224×224 + ImageNet norm; serve must match — same shared lib or container image version as training export.
Batching tradeoffs. Images vary in size — dynamic padding; GPU memory spikes with batch. Often batch size 1 for interactive API.
Latency budget decomposition.
Download image: 20–200 ms
Preprocess: 10–50 ms
Forward pass: 100 ms – several s
Token generation: variable
Set client timeout accordingly.
Storage & privacy. Medical/retail images — encrypt at rest, short TTL on temp buckets, no logging raw bytes.
Rate limiting. Multimodal abuse ( huge images ) → max dimension validation at gateway.
Evaluation in prod. Different from tabular: sample human review, toxicity checks, hallucination monitoring — product metrics beyond RED.
Как это выглядит на практике
API design (sketch):
POST /v1/vision/analyze { "image_url": "https://minio.example.com/temp-uploads/abc123.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature=...", "prompt": "List product defects visible.", "max_tokens": 256 }
Gateway принимает только HTTPS presigned URL с коротким TTL; transformer скачивает его и валидирует magic bytes. Это также защита от SSRF: allowlist разрешённых host'ов object storage, запрет private/link-local IP и повторная проверка адреса после DNS resolution.
KServe layout:
transformer container: download + decode image + build model-specific tensors.
predictor vLLM/Triton with VLM weights.
Capacity. One VLM instance per GPU common; scale replicas not batch for interactive.
Course scope. MDP may provide demo VLM endpoint read-only; capstone tabular teams — understand pattern for future, not mandatory implement.
Что сделать после занятия
Почему presigned URL лучше base64 для prod?
Нарисуйте latency budget для одного VLM запроса (5 компонентов).
Назовите 2 риска логирования multimodal requests.