Principle:Kserve Kserve Multi Model Shared Service
| Knowledge Sources | |
|---|---|
| Domains | MLOps, Cost_Optimization, Model_Serving |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
A resource efficiency pattern that deploys a shared InferenceService pod to host multiple models, reducing per-model infrastructure overhead.
Description
Multi-Model Shared Service enables multiple ML models to be served from a single InferenceService pod. Instead of deploying N separate pods for N models, one pod with a model agent sidecar dynamically loads and unloads models based on TrainedModel resources.
The key differentiator is the absence of storageUri in the InferenceService spec. When IsMMSPredictor() returns true (no storageUri, not HuggingFace), the controller creates an empty ConfigMap (modelconfig-<name>-0) and enables the model agent sidecar, which watches for TrainedModel-driven model configurations.
Usage
Use this when serving many small-to-medium models that individually would waste resources. Ideal for:
- Sklearn/XGBoost models that share a runtime
- High model count with low per-model traffic
- Cost-sensitive deployments where per-pod overhead matters
Theoretical Basis
# MMS detection logic (NOT implementation code)
IsMMSPredictor(predictor) → bool:
return predictor.storageUri == "" AND
predictor.storageSpec == nil AND
predictor.storageUris == nil AND
predictor.modelFormat.name != "huggingface"
When MMS mode:
1. Create InferenceService WITHOUT storageUri
2. Controller creates empty ConfigMap: modelconfig-<name>-0
3. Agent sidecar injected into pod
4. TrainedModel resources populate ConfigMap
5. Agent detects ConfigMap changes → downloads and loads models