Implementation:Tensorflow Serving Bundle Factory Util

Knowledge Sources	Tensorflow_Serving
Domains	Model Serving, Configuration, Batching
Last Updated	2026-02-13 00:00 GMT

Overview

Provides utility functions for configuring TensorFlow session bundles, including session/run options extraction, batch scheduler creation, resource estimation, and session wrapping for batching or read-only access.

Description

The Bundle Factory Util module is a collection of utility functions that support SavedModel bundle loading and configuration. Key functions include:

• GetSessionOptions: Extracts SessionOptions from a SessionBundleConfig protobuf, setting the session target and config.
• GetRunOptions: Extracts RunOptions from a SessionBundleConfig, configuring the inter-op thread pool index if specified.
• GetPerModelBatchingParams: Resolves per-model batching parameters by checking for a batching_params.pbtxt file in the model directory when per_model_configured is true, or returning common_params otherwise.
• CreateBatchScheduler (template): Creates a SharedBatchScheduler from BatchingParameters, configuring thread count and thread pool name.
• GetQueueOptions (template): Constructs queue options from BatchingParameters, including max_batch_size, batch_timeout_micros, max_enqueued_batches, and large batch splitting configuration with a custom split_input_task_func.
• EstimateResourceFromValidationResult: Estimates RAM usage from infrastructure validation results.
• EstimateResourceFromPath: Estimates RAM from the model path, optionally using validation results or a disk-based heuristic.
• WrapSessionForBatching: Wraps a TensorFlow Session with a batching layer using a SharedBatchScheduler, configuring allowed batch sizes and variable-length input padding. Validates that the last allowed batch size matches the maximum batch size.
• WrapSession: Wraps a Session in a ServingSessionWrapper that blocks state-changing methods.
• WrapSessionIgnoreThreadPoolOptions: Wraps a Session in a wrapper that ignores thread pool options (for RemoteSession compatibility).

Usage

Use these utilities when setting up model serving infrastructure. They are used by both the SavedModelBundleFactory and TfrtSavedModelFactory for common configuration tasks like batch scheduling, resource estimation, and session wrapping.

Code Reference

Source Location

• Repository: Tensorflow_Serving
• Files:
  • tensorflow_serving/servables/tensorflow/bundle_factory_util.h (lines 1-143)
  • tensorflow_serving/servables/tensorflow/bundle_factory_util.cc (lines 1-182)

Signature

SessionOptions GetSessionOptions(const SessionBundleConfig& config);
RunOptions GetRunOptions(const SessionBundleConfig& config);

Status GetPerModelBatchingParams(const string& path,
                                 const BatchingParameters& common_params,
                                 bool per_model_configured,
                                 absl::optional<BatchingParameters>* params);

template <typename TaskType>
Status CreateBatchScheduler(
    const BatchingParameters& batching_config,
    std::shared_ptr<SharedBatchScheduler<TaskType>>* batch_scheduler);

Status WrapSessionForBatching(
    const BatchingParameters& batching_config,
    std::shared_ptr<SharedBatchScheduler<BatchingSessionTask>> batch_scheduler,
    const std::vector<SignatureDef>& signatures,
    std::unique_ptr<Session>* session);

Status WrapSession(std::unique_ptr<Session>* session);
Status WrapSessionIgnoreThreadPoolOptions(std::unique_ptr<Session>* session);

Import

#include "tensorflow_serving/servables/tensorflow/bundle_factory_util.h"

I/O Contract

Inputs

Name	Type	Required	Description
config	SessionBundleConfig	Yes	Configuration protobuf for session and run options
batching_config	BatchingParameters	Yes	Batching configuration with thread counts, batch sizes, and timeout settings
session	std::unique_ptr<Session>*	Yes	Session to wrap (for wrapping functions)
path	string	Yes	Model export path (for resource estimation and per-model batching)

Outputs

Name	Type	Description
session	std::unique_ptr<Session>*	Wrapped session (modified in place)
estimate	ResourceAllocation*	Estimated resource usage
return	Status	OK on success; InvalidArgument for configuration errors

Usage Examples

Configuring Batching for a Session

BatchingParameters batching_config;
batching_config.mutable_max_batch_size()->set_value(128);
batching_config.mutable_num_batch_threads()->set_value(4);

std::shared_ptr<SharedBatchScheduler<BatchingSessionTask>> scheduler;
TF_RETURN_IF_ERROR(CreateBatchScheduler(batching_config, &scheduler));

std::vector<SignatureDef> signatures = {signature_def};
TF_RETURN_IF_ERROR(WrapSessionForBatching(
    batching_config, scheduler, signatures, &session));

Related Pages

• Principle:Tensorflow_Serving_Bundle_Factory_Configuration