Implementation:Tensorflow Serving Caching Manager Test

Knowledge Sources	Tensorflow_Serving
Domains	Testing, Core Management
Last Updated	2026-02-13 00:00 GMT

Overview

Test suite validating the CachingManager which loads servables on-demand and caches them for subsequent requests.

Description

This test file exercises the CachingManager class, a manager that lazily loads servables upon first request and caches them for reuse. The tests use two custom LoaderFactory implementations: StringLoaderFactory (produces string servables by concatenating name and version) and ErrorLoaderFactory (always returns error loaders). The CachingManagerTest fixture is parameterized on thread pool sizes for testing both inline and threaded execution. Tests cover single and multiple requests, earliest/latest version policies, wrong-type handles, error handling, available handle listing, event bus integration, and concurrent request handling (both disjoint and intersecting). Additional non-parameterized tests validate the PathPrefixLoaderFactory.

Usage

Run these tests to verify that the caching manager correctly loads servables on demand, caches results, handles concurrent requests efficiently, and properly integrates with the event bus. Important after changes to caching logic, loader factories, or version resolution policies.

Code Reference

Source Location

Repository: Tensorflow_Serving
File: tensorflow_serving/core/caching_manager_test.cc
Lines: 1-618

Test Fixture

// A simple loader-factory that concatenates requested servable name and version.
class StringLoaderFactory : public CachingManager::LoaderFactory {
 public:
  explicit StringLoaderFactory(const int64_t starting_version)
      : latest_version_(starting_version) {}
  ~StringLoaderFactory() override = default;

  ServableData<std::unique_ptr<Loader>> CreateLoader(
      const ServableId& id) override {
    {
      mutex_lock l(mu_);
      num_loaders_dispensed_++;
    }
    auto servable_creator = [&](std::unique_ptr<string>* servable) {
      servable->reset(new string);
      **servable = absl::StrCat(id.name, "-", id.version);
      return absl::OkStatus();
    };
    std::unique_ptr<Loader> loader;
    loader.reset(new SimpleLoader<string>(
        servable_creator, SimpleLoader<string>::EstimateNoResources()));
    return ServableData<std::unique_ptr<Loader>>(id, std::move(loader));
  }

  int64_t GetServableVersion(
      const string& request_name,
      ServableRequest::AutoVersionPolicy policy) const override;

  void set_earliest_version(int64_t version);
  void set_latest_version(int64_t version);
  int64_t num_loaders_dispensed() const;

 private:
  mutable mutex mu_;
  int64_t earliest_version_ TF_GUARDED_BY(mu_) = 0;
  int64_t latest_version_ TF_GUARDED_BY(mu_) = 0;
  int64_t num_loaders_dispensed_ TF_GUARDED_BY(mu_) = 0;
};

Build Target

bazel test //tensorflow_serving/core:caching_manager_test

Test Coverage

Key Test Cases

Test Name	Category	Description
`ServableHandleSingleRequest`	Core	Validates a single servable handle request loads and caches correctly
`ServableHandleMultipleRequests`	Caching	Ensures multiple requests for the same servable only create one loader
`ServableHandleSingleRequestEarliest`	Version Policy	Tests earliest version policy for handle retrieval
`ServableHandleSingleRequestLatest`	Version Policy	Tests latest version policy for handle retrieval
`ServableHandleMultipleRequestsEarliest`	Version Policy	Multiple requests with earliest version policy
`ServableHandleMultipleRequestsLatest`	Version Policy	Multiple requests with latest version policy (version changes)
`ServableHandleWrongType`	Validation	Verifies INVALID_ARGUMENT when requesting wrong handle type
`ServableHandleError`	Error Handling	Tests error propagation from ErrorLoaderFactory
`AvailableServableHandlesNoRequests`	Listing	Empty handles map when no requests have been made
`AvailableServableHandlesMultipleRequests`	Listing	Correct handles map after multiple load requests
`AvailableServableHandlesWrongType`	Listing	Empty map when requesting handles with wrong type
`AvailableServableHandlesError`	Error Handling	Handles map excludes errored servables
`ListAvailableServableIdsMultipleRequests`	Listing	Lists IDs after multiple distinct requests
`EventBusSingleRequest`	Events	Validates event bus notifications on single load
`EventBusErrorHandle`	Events	Validates event bus error notifications
`ConcurrentDisjointRequests`	Concurrency	Tests concurrent requests for different servables
`ConcurrentIntersectingRequests`	Concurrency	Tests concurrent requests for the same servable (only one load)
`PathPrefixLoaderFactoryBasic`	Factory	Tests PathPrefixLoaderFactory basic functionality
`PathPrefixLoaderFactoryVersionOtherThanZeroYieldsError`	Factory	Validates PathPrefixLoaderFactory rejects non-zero versions

Usage Examples

Test Pattern

// A simple loader-factory that always returns a loader with an error.
class ErrorLoaderFactory : public CachingManager::LoaderFactory {
 public:
  ErrorLoaderFactory() = default;
  ~ErrorLoaderFactory() override = default;

  ServableData<std::unique_ptr<Loader>> CreateLoader(
      const ServableId& id) override {
    auto servable_creator = [&](std::unique_ptr<string>* servable) {
      return errors::Unknown("error loader-factory");
    };
    std::unique_ptr<Loader> loader;
    loader.reset(new SimpleLoader<string>(
        servable_creator, SimpleLoader<string>::EstimateNoResources()));
    return ServableData<std::unique_ptr<Loader>>(id, std::move(loader));
  }

  int64_t GetServableVersion(
      const string& request_name,
      ServableRequest::AutoVersionPolicy policy) const override {
    return 42;
  }
};

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment