Implementation:Tensorflow Serving caching manager h

Knowledge Sources	Tensorflow_Serving
Domains	Model Serving, Core Framework
Last Updated	2026-02-13 00:00 GMT

Overview

CachingManager is a Manager implementation that loads servables on-demand upon first request, caching them for subsequent access via a wrapped BasicManager.

Description

CachingManager extends Manager and provides an on-demand loading strategy: when a servable is requested, the manager checks if it is already loaded, and if not, initiates a synchronous load operation before returning the handle. This contrasts with the push-based AspiredVersionsManager, which loads servables proactively based on source notifications.

The manager wraps a BasicManager for actual servable lifecycle management and delegates to a pluggable LoaderFactory to create loaders for requested servables. The LoaderFactory interface has two methods:

• CreateLoader() - Creates a ServableData<std::unique_ptr<Loader>> for a given ServableId.
• GetServableVersion() - Returns a version number for requests that do not specify an explicit version.

For concurrent requests for the same servable, a per-servable mutex map ensures that exactly one thread performs the load while others block. The mutex map uses reference-counted shared_ptr<mutex> entries that are garbage-collected when no longer needed.

The header also defines PathPrefixLoaderFactory, a simple LoaderFactory that constructs storage paths by concatenating a fixed prefix with the servable name, supporting only single-version servables at version 0.

Configuration is provided via the Options struct, which mirrors BasicManager::Options (resource tracker, thread counts, retry settings, environment, event bus).

Usage

Use CachingManager when servables should be loaded lazily on first access rather than proactively. This is suitable for scenarios with a large number of potential servables where only a subset is actively used, or when the full set of servables is not known at startup.

Code Reference

Source Location

• Repository: Tensorflow_Serving
• File: tensorflow_serving/core/caching_manager.h
• Lines: 1-193

Signature

class CachingManager : public Manager {
 public:
  struct Options {
    std::unique_ptr<ResourceTracker> resource_tracker;
    uint32 num_load_threads = 0;
    uint32 num_unload_threads = 0;
    EventBus<ServableState>* servable_event_bus = nullptr;
    uint32 max_num_load_retries = 5;
    int64_t load_retry_interval_micros = 1LL * 60 * 1000 * 1000;
    Env* env = Env::Default();
  };

  class LoaderFactory {
   public:
    virtual ~LoaderFactory() = default;
    virtual ServableData<std::unique_ptr<Loader>> CreateLoader(
        const ServableId& servable_id) = 0;
    virtual int64_t GetServableVersion(
        const string& servable_name,
        ServableRequest::AutoVersionPolicy policy) const = 0;
  };

  static Status Create(Options options,
                       std::unique_ptr<LoaderFactory> loader_factory,
                       std::unique_ptr<CachingManager>* caching_manager);
  ~CachingManager() override;

  std::map<ServableId, std::unique_ptr<UntypedServableHandle>>
  GetAvailableUntypedServableHandles() const override;
  std::vector<ServableId> ListAvailableServableIds() const override;
};

class PathPrefixLoaderFactory : public CachingManager::LoaderFactory {
 public:
  PathPrefixLoaderFactory(const string& path_prefix,
                          std::unique_ptr<StoragePathSourceAdapter> adapter);
  ServableData<std::unique_ptr<Loader>> CreateLoader(const ServableId& id) override;
  int64_t GetServableVersion(const string& servable_name,
                             ServableRequest::AutoVersionPolicy policy) const override;
};

Import

#include "tensorflow_serving/core/caching_manager.h"

I/O Contract

Inputs

Name	Type	Required	Description
options	Options	Yes	Configuration struct with thread counts, retry settings, resource tracker, etc.
loader_factory	std::unique_ptr<LoaderFactory>	Yes	Factory for creating loaders on-demand from servable ids
request	ServableRequest	Yes (for GetServableHandle)	Request specifying the servable name and optional version

Outputs

Name	Type	Description
Create()	Status	OK on success; constructs the CachingManager
GetUntypedServableHandle()	Status	Returns a handle after loading (if needed); error on load failure
GetAvailableUntypedServableHandles()	std::map<ServableId, unique_ptr<UntypedServableHandle>>	All currently loaded servable handles
ListAvailableServableIds()	std::vector<ServableId>	All currently loaded servable identifiers

Usage Examples

Creating a CachingManager with PathPrefixLoaderFactory

#include "tensorflow_serving/core/caching_manager.h"

using namespace tensorflow::serving;

CachingManager::Options options;
options.num_load_threads = 4;
options.max_num_load_retries = 3;

auto adapter = std::make_unique<MyStoragePathSourceAdapter>();
auto factory = std::make_unique<PathPrefixLoaderFactory>(
    "/models/", std::move(adapter));

std::unique_ptr<CachingManager> manager;
TF_CHECK_OK(CachingManager::Create(
    std::move(options), std::move(factory), &manager));

// First request triggers on-demand load
ServableHandle<MyModel> handle;
TF_CHECK_OK(manager->GetServableHandle(
    ServableRequest::FromId({"my_model", 0}), &handle));

// Subsequent requests use cached servable
ServableHandle<MyModel> handle2;
TF_CHECK_OK(manager->GetServableHandle(
    ServableRequest::FromId({"my_model", 0}), &handle2));

Related Pages

• Principle:Tensorflow_Serving_Servable_Caching