Implementation:Tensorflow Serving CreateBasicBatchingSession

Knowledge Sources	TensorFlow Serving
Domains	Performance, Scheduling
Last Updated	2026-02-13 17:00 GMT

Overview

Concrete tool for wrapping a TensorFlow Session with transparent request batching using a BasicBatchScheduler, provided by the batching_session module.

Description

CreateBasicBatchingSession() is a convenience function that wraps a TensorFlow Session with a BatchingSession using a BasicBatchScheduler. It creates a single scheduler for one tensor signature.

For multiple signatures, CreateBatchingSession() accepts a list of signature-scheduler pairs. The BatchingSession internally:

InternalRun() (L376-458): Creates BatchingSessionTask, schedules it, waits for completion
ProcessBatch() (L715-818): Merges inputs, runs session, splits outputs
MergeInputTensors() (L486-591): Concatenates along 0th dimension
SplitOutputTensors() (L593-678): Slices outputs back to individual responses

Usage

Created internally by the serving infrastructure when enable_batching=true. Users configure behavior via BasicBatchScheduler::Options parameters.

Code Reference

Source Location

Repository: tensorflow/serving
File: tensorflow_serving/batching/batching_session.cc
Lines: L1005-1073 (CreateBasicBatchingSession), L976-1003 (CreateBatchingSession), L314-342 (BatchingSession::Create)
Header: tensorflow_serving/batching/batching_session.h L130-154

Signature

Status CreateBasicBatchingSession(
    const BasicBatchScheduler<BatchingSessionTask>::Options& schedule_options,
    const BatchingSessionOptions& batching_session_options,
    const TensorSignature& signature,
    std::unique_ptr<Session> session,
    std::unique_ptr<Session>* batching_session
);

Status CreateBatchingSession(
    const BatchingSessionOptions& options,
    const std::vector<SignatureWithBatchingSessionSchedulerCreator>&
        signatures_with_scheduler_creators,
    std::unique_ptr<Session> session,
    std::unique_ptr<Session>* batching_session
);

Import

#include "tensorflow_serving/batching/batching_session.h"

I/O Contract

Inputs

Name	Type	Required	Description
schedule_options.max_batch_size	int	Yes	Maximum items per batch
schedule_options.batch_timeout_micros	int64	Yes	Max wait time before processing partial batch
schedule_options.num_batch_threads	int	Yes	Threads for batch processing
schedule_options.max_enqueued_batches	int	Yes	Max queue depth
schedule_options.enable_large_batch_splitting	bool	No	Split oversized tasks
batching_session_options.allowed_batch_sizes	vector<int>	No	Restricted batch sizes for padding
batching_session_options.pad_variable_length_inputs	bool	No	Pad variable-length tensors
signature	TensorSignature	Yes	Input/output tensor names
session	unique_ptr<Session>	Yes	Original unwrapped session

Outputs

Name	Type	Description
batching_session	unique_ptr<Session>*	Session that transparently batches Run() calls

Usage Examples

C++ Usage

#include "tensorflow_serving/batching/batching_session.h"

// Configure scheduler options
BasicBatchScheduler<BatchingSessionTask>::Options schedule_options;
schedule_options.max_batch_size = 128;
schedule_options.batch_timeout_micros = 10000;  // 10ms
schedule_options.num_batch_threads = 4;
schedule_options.max_enqueued_batches = 1000;

// Configure batching session options
BatchingSessionOptions batching_options;
batching_options.allowed_batch_sizes = {8, 16, 32, 64, 128};

// Wrap the session
std::unique_ptr<Session> batching_session;
TF_CHECK_OK(CreateBasicBatchingSession(
    schedule_options, batching_options, signature,
    std::move(session), &batching_session));

// Use batching_session just like a normal session
// Run() calls are now transparently batched

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment