Principle:Tensorflow Serving Bundle Factory Configuration
| Knowledge Sources | |
|---|---|
| Domains | Model Serving, Configuration, Batching |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
Bundle Factory Configuration defines the utility patterns for extracting session/run options, configuring batch schedulers, estimating resources, and wrapping sessions for safe serving use.
Description
The Bundle Factory Configuration principle governs the shared configuration utilities used by model bundle factories (both SavedModelBundleFactory and TfrtSavedModelFactory). These utilities translate protobuf-based configuration into runtime objects and enforce operational safety constraints.
Key patterns:
Configuration Extraction: Session and run options are extracted from the SessionBundleConfig protobuf, providing a centralized mapping from configuration to TensorFlow runtime objects.
Per-Model Batching: Models can override global batching parameters by placing a batching_params.pbtxt file in their export directory. This enables model-specific tuning (e.g., different batch sizes for models with different computational profiles) while maintaining a common default.
Batch Scheduler Creation: Template functions create SharedBatchScheduler instances and configure queue options (batch size limits, timeouts, enqueued batch limits, large batch splitting) from BatchingParameters protobufs.
Session Wrapping: Sessions are wrapped in safety layers: ServingSessionWrapper blocks state-changing operations, and SessionWrapperIgnoreThreadPoolOptions provides RemoteSession compatibility.
Resource Estimation: Model resource requirements are estimated from either validation results or disk-based heuristics, enabling capacity planning before loading models.
Usage
Apply this principle when adding new configuration options to the model loading pipeline. Configuration should be extracted from protobufs using centralized utility functions, batching should support per-model overrides, and sessions should always be wrapped for serving safety.
Theoretical Basis
The configuration pattern follows the principle of separating policy (what to configure) from mechanism (how to configure). Protobuf-based configuration provides:
- Serializable configuration: Parameters can be stored, transmitted, and versioned.
- Validation at parse time: Protobuf types enforce basic structure and type safety.
- Defaults and overrides: Per-model configuration overrides global defaults, implementing a hierarchical configuration pattern.
The session wrapping pattern follows the Decorator design pattern, adding cross-cutting concerns (safety, batching, thread pool management) without modifying the underlying session implementation.