Principle:Kserve Kserve Model Agent Loading

Knowledge Sources	KServe Multi-Model Serving
Domains	MLOps, Model_Management, Storage
Last Updated	2026-02-13 00:00 GMT

Overview

A sidecar agent pipeline that watches for model configuration changes, downloads model artifacts, and loads them into the model server via the V2 repository API.

Description

The Model Agent Loading pipeline is a three-stage sidecar process:

• Watcher: Monitors the ConfigMap mount directory for changes using fsnotify. When the Kubernetes ..data symlink is updated, it parses the new config and computes a three-way diff (added, updated, removed models).
• Puller: Processes model operations asynchronously. For added models, it downloads artifacts then calls the V2 repository load API. For removed models, it deletes local files then calls the V2 unload API.
• Downloader: Downloads model files from cloud storage (S3, GCS, HTTP). Uses SHA256-based idempotency via SUCCESS.<sha256> marker files to avoid redundant downloads.

Usage

This pipeline runs automatically in every MMS pod. Understanding it is important for debugging model loading failures or optimizing download performance.

Theoretical Basis

# Agent pipeline (NOT implementation code)
1. Watcher detects ConfigMap ..data symlink change
2. parseConfig() computes diff:
   - New models → modelAdded(name, spec)
   - Changed models → modelRemoved(old) + modelAdded(new)
   - Stale models → modelRemoved(name)
3. ModelOp{Op: Add/Remove} sent to Puller channel
4. Puller.modelProcessor() per model:
   Add:
     a. Downloader.DownloadModel(uri, localPath)
     b. POST /v2/repository/models/{name}/load
   Remove:
     a. RemoveDir(localPath)
     b. POST /v2/repository/models/{name}/unload
5. Downloader checks SUCCESS.<sha256> for idempotency

Related Pages

Implemented By

• Implementation:Kserve_Kserve_Model_Agent_Pipeline