Implementation:Tensorflow Serving Thread Pool Factory

Knowledge Sources	Tensorflow_Serving
Domains	Model Serving, Thread Management
Last Updated	2026-02-13 00:00 GMT

Overview

Defines the abstract ThreadPoolFactory interface and ScopedThreadPools helper for managing custom inter-op and intra-op thread pools used during TensorFlow inference execution.

Description

The Thread Pool Factory module provides an abstraction layer for custom thread pool management in TensorFlow Serving:

ScopedThreadPools is a utility class that holds shared_ptr references to inter-op and intra-op thread pool interfaces and returns them as tensorflow::thread::ThreadPoolOptions. The shared ownership ensures that thread pools remain alive for the lifetime of the ScopedThreadPools instance. The default constructor sets both thread pools to nullptr, which causes the TensorFlow runtime to use its default thread pools.

ThreadPoolFactory is an abstract base class with a single pure virtual method GetThreadPools() that returns a ScopedThreadPools instance. Concrete implementations create and manage specific thread pool configurations (e.g., custom pool sizes, priorities, or allocation strategies).

The module also defines a ThreadPoolFactoryRegistry using the class registration macro system (DEFINE_CLASS_REGISTRY / REGISTER_CLASS), allowing thread pool factory implementations to be registered with their configuration protobuf types and created dynamically from configuration files.

Usage

Use this module when custom thread pool management is needed for TensorFlow Serving. Thread pool factories are configured via thread_pool_factory_config_filepath in TfrtSavedModelConfig and created by the TfrtSavedModelFactory. The factory is shared across all servables created by a given model factory. Implement a concrete ThreadPoolFactory subclass and register it with REGISTER_THREAD_POOL_FACTORY to provide custom thread pool behavior.

Code Reference

Source Location

• Repository: Tensorflow_Serving
• File: tensorflow_serving/servables/tensorflow/thread_pool_factory.h (lines 1-63)

Signature

class ScopedThreadPools {
 public:
  ScopedThreadPools() = default;
  ScopedThreadPools(
      std::shared_ptr<thread::ThreadPoolInterface> inter_op_thread_pool,
      std::shared_ptr<thread::ThreadPoolInterface> intra_op_thread_pool);
  tensorflow::thread::ThreadPoolOptions get();
};

class ThreadPoolFactory {
 public:
  virtual ~ThreadPoolFactory() = default;
  virtual ScopedThreadPools GetThreadPools() = 0;
};

DEFINE_CLASS_REGISTRY(ThreadPoolFactoryRegistry, ThreadPoolFactory);
#define REGISTER_THREAD_POOL_FACTORY(ClassCreator, ConfigProto)

Import

#include "tensorflow_serving/servables/tensorflow/thread_pool_factory.h"

I/O Contract

Inputs

Name	Type	Required	Description
inter_op_thread_pool	shared_ptr<thread::ThreadPoolInterface>	No	Custom inter-op thread pool (nullptr for default)
intra_op_thread_pool	shared_ptr<thread::ThreadPoolInterface>	No	Custom intra-op thread pool (nullptr for default)

Outputs

Name	Type	Description
GetThreadPools()	ScopedThreadPools	Thread pool options instance with scoped lifetime management
get()	thread::ThreadPoolOptions	TensorFlow thread pool options with raw pointers to the managed pools

Usage Examples

Using ThreadPoolFactory with a Servable

// ThreadPoolFactory is typically used via TfrtSavedModelFactory
auto thread_pool_options =
    thread_pool_factory_ == nullptr
        ? tsl::thread::ThreadPoolOptions()
        : thread_pool_factory_->GetThreadPools().get();

// Pass to prediction
internal::RunPredict(run_options, version, option,
                     saved_model, request, &response,
                     thread_pool_options);

Related Pages

• Principle:Tensorflow_Serving_Thread_Pool_Management