7.6.2 · блок 7
VLM и multimodal serving
VLM и multimodal serving
Зачем это нужно
Vision-Language Models (VLM) принимают изображение + текст («что на фото?», OCR, visual QA). Payload больше, preprocessing сложнее, latency выше, чем у tabular ML. MLOps-инженер должен понимать multimodal inference path: upload → storage → resize/tokenize → GPU forward → response.
Урок расширяет serving stack на CV+LLM без глубокого DL — фокус на операционных отличиях.
Основные идеи
Multimodal request anatomy.
Client → API Gateway
├── image (base64 or presigned URL)
├── text prompt
└── optional parameters (max_tokens, temperature)
→ Preprocessor (resize, normalize, patch embed)
→ VLM model (GPU)
→ Text (and optionally structured JSON) response
Payload size problem. 5 MB image in JSON base64 — bad for Kafka/gateway limits. Pattern: presigned URL to object storage (MinIO); inference Pod downloads internally.
Runtimes for VLM.
| Stack | Notes |
|-------|-------|
| vLLM | Growing multimodal support (model-dependent) |
| Triton + Python/ONNX | Custom pre/post in backend |
| Dedicated (LMDeploy, etc.) | Alternative engines |
Check model card for supported server.
Preprocessing consistency. Train used 224×224 + ImageNet norm; serve must match — same shared lib or container image version as training export.
Batching tradeoffs. Images vary in size — dynamic padding; GPU memory spikes with batch. Often batch size 1 for interactive API.
Latency budget decomposition.
- Download image: 20–200 ms
- Preprocess: 10–50 ms
- Forward pass: 100 ms – several s
- Token generation: variable
Set client timeout accordingly.
Storage & privacy. Medical/retail images — encrypt at rest, short TTL on temp buckets, no logging raw bytes.
Rate limiting. Multimodal abuse ( huge images ) → max dimension validation at gateway.
Evaluation in prod. Different from tabular: sample human review, toxicity checks, hallucination monitoring — product metrics beyond RED.
Как это выглядит на практике
API design (sketch):
POST /v1/vision/analyze
{
"image_url": "https://minio.example.com/temp-uploads/abc123.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature=...",
"prompt": "List product defects visible.",
"max_tokens": 256
}
Gateway принимает только HTTPS presigned URL с коротким TTL; transformer скачивает его и валидирует magic bytes. Это также защита от SSRF: allowlist разрешённых host'ов object storage, запрет private/link-local IP и повторная проверка адреса после DNS resolution.
KServe layout:
- transformer container: download + decode image + build model-specific tensors.
- predictor vLLM/Triton with VLM weights.
Capacity. One VLM instance per GPU common; scale replicas not batch for interactive.
Course scope. MDP may provide demo VLM endpoint read-only; capstone tabular teams — understand pattern for future, not mandatory implement.
Что сделать после занятия
- [ ] Почему presigned URL лучше base64 для prod?
- [ ] Нарисуйте latency budget для одного VLM запроса (5 компонентов).
- [ ] Назовите 2 риска логирования multimodal requests.