Heuristic:Tensorflow Serving Batching Thread Tuning
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Batching, ML_Serving |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
Set client threads to 2x the maximum batch size to achieve optimal throughput with batching enabled, with distinct tuning strategies for CPU-only vs GPU workloads.
Description
TensorFlow Serving's `BatchingSession` operates synchronously: each `Session::Run()` call blocks waiting for enough peer calls to form a full batch. This means client threads spend most of their time waiting, not computing. If the thread count is too low, batches never fill and throughput suffers. If too high, excess threads waste resources without benefit. The empirically-derived rule is to set client threads to approximately twice the maximum batch size. For multi-signature models, use twice the sum of all signatures' maximum batch sizes. CPU and GPU workloads require fundamentally different tuning approaches.
Usage
Use this heuristic when configuring batching parameters for TensorFlow Serving, especially when you observe low throughput despite batching being enabled. This applies to both the global `--batching_parameters_file` configuration and per-model batching parameters.
The Insight (Rule of Thumb)
- Action: Set the number of client threads to `2 * max_batch_size` (for single-signature models) or `2 * SUM(max_batch_size per signature)` (for multi-signature models).
- Value: Thread count = 2x batch size.
- Trade-off: Too few threads means batches never fill (low throughput). Too many threads waste memory and context-switching overhead.
CPU-Only Tuning Strategy
- Set `num_batch_threads` equal to the number of CPU cores
- Set `max_batch_size` to a large value
- Start with `batch_timeout_micros` = 0
- Experiment with values in the 1-10 ms (1000-10000 microsecond) range
- Note: 0 may be the optimal value for `batch_timeout_micros` on CPU
GPU Tuning Strategy
- Set `num_batch_threads` to the number of CPU cores
- Temporarily set `batch_timeout_micros` to a large value while tuning `max_batch_size` to balance throughput vs latency. Consider values in the hundreds or thousands.
- For online serving, tune `batch_timeout_micros` to control tail latency. Best values are typically a few milliseconds. Zero is worth testing.
allowed_batch_sizes Guidance
- Entries must be in increasing order
- Final entry must equal `max_batch_size`
- Consider exponential sequences: `[8, 16, 32, ..., max]`
- Or linear sequences: `[100, 200, 300, ..., max]`
- Or hybrid: `[8, 16, 32, 64, 100, 200, 300, ..., max]`
Reasoning
`BatchingSession` is synchronous: each `Session::Run()` blocks until enough peers arrive to form a batch. With N max batch size, you need approximately 2N threads to ensure batches fill while some threads are in the run/return phase. The batching README explains that hardware accelerators (GPUs) require batching to unlock their throughput potential. The 2x multiplier accounts for the round-trip time: while one batch is executing, the next batch's worth of threads should be accumulating.
For GPU workloads, the tuning is multi-step because GPU utilization depends on batch size (too small = GPU underutilized, too large = latency increases). For CPU workloads, the relationship is simpler because CPUs don't have the same batch-level parallelism gains.
The `batch_timeout_micros` = 0 with `StreamingBatchScheduler` produces single-item batches (effectively disabling batching), which is different from "process immediately when a thread is available". This is a common pitfall.
Code Evidence
Thread count recommendation from `batching_session.h:104-108`:
// IMPORTANT: Each call to Session::Run() is synchronous, and blocks waiting for
// other Run() calls with the same signature to merge with to form a large
// batch. Consequently, to achieve good throughput we recommend setting the
// number of client threads that call Session::Run() equal to about twice the
// sum over all signatures of the maximum batch size.
Thread multiplier rule from `session_bundle_config.proto:156-158`:
// IMPORTANT: As discussed above, use 'max_batch_size * 2' client threads to
// achieve high throughput with batching.
CPU-only tuning from `batching/README.md:143-150`:
If your system is CPU-only (no GPU), then consider starting with the following
values: `num_batch_threads` equal to the number of CPU cores; `max_batch_size`
to a really high value; `batch_timeout_micros` to 0. Then experiment with
`batch_timeout_micros` values in the 1-10 millisecond (1000-10000 microsecond)
range, while keeping in mind that 0 may be the optimal value.
Batch timeout zero pitfall from `streaming_batch_scheduler.h:131-136`:
// Setting this value to 0 will *not* result in the behavior of processing
// a batch as soon as a thread becomes available. Instead, it will cause
// each batch to contain just a single item, essentially disabling batching.
// StreamingBatchScheduler is not the right vehicle for achieving the
// aforementioned behavior.
Related Pages
- Implementation:Tensorflow_Serving_Batching_CLI_Configuration
- Implementation:Tensorflow_Serving_CreateBasicBatchingSession
- Implementation:Tensorflow_Serving_BatchingOptions_Configuration
- Implementation:Tensorflow_Serving_StreamingBatchScheduler
- Principle:Tensorflow_Serving_Batching_Enablement
- Principle:Tensorflow_Serving_Batch_Scheduling_Configuration
- Principle:Tensorflow_Serving_Hardware_Tuning