Principle:Tensorflow Serving Servable Access Pattern

Knowledge Sources	Tensorflow_Serving
Domains	Model Serving, Core Framework
Last Updated	2026-02-13 00:00 GMT

Overview

The Servable Access Pattern principle defines how frontend code safely obtains typed, reference-counted access to loaded servable objects through smart-pointer handles.

Description

When a client request arrives, the serving frontend needs to obtain a reference to the loaded servable (e.g., a TensorFlow SavedModel) to perform inference. This access must be:

• Type-safe: The handle must provide access to the correct concrete type without manual casting.
• Lifecycle-safe: The servable must remain alive for the duration of the request, even if a newer version is being loaded.
• Non-blocking for version transitions: Handles should be held briefly so that the system can transition to new servable versions without delay.

The pattern uses a two-layer handle design:

1. UntypedServableHandle - An internal, type-erased handle used by the Manager. It provides access to the servable via AnyPtr (a type-safe void pointer).
2. ServableHandle<T> - A typed wrapper obtained by frontend code. It performs the type-safe downcast from AnyPtr and provides smart-pointer semantics (operator->, operator*).

The SharedPtrHandle implementation uses shared_ptr<Loader> for reference counting, ensuring the Loader (and thus the servable) stays alive as long as any handle exists.

Usage

Apply this principle whenever frontend code needs to access a loaded servable. Obtain a ServableHandle<T> from the Manager, use it for the duration of the request, and release it promptly. Never cache handles across requests, as this can prevent servable version transitions.

Theoretical Basis

The handle pattern implements a type-erased reference-counted smart pointer with two levels of abstraction:

Manager -> UntypedServableHandle (type-erased, ref-counted via shared_ptr)
         -> ServableHandle<T> (typed wrapper, smart-pointer interface)

Lifecycle guarantee:
  While handle exists -> shared_ptr<Loader> alive -> servable alive
  Handle released -> ref count decremented -> version transition unblocked

Key design properties:

• Type erasure at the Manager boundary: Managers work with UntypedServableHandle to avoid being templatized on servable types.
• Type recovery at the frontend: ServableHandle<T> recovers the type via AnyPtr::get<T>(), failing at runtime if the type does not match.
• Pointer prohibition: The static_assert preventing T* as the template argument prevents double-indirection mistakes.
• Equality includes identity: Handle comparison checks both the pointer and the ServableId, distinguishing handles to the same servable object in different version slots.

Related Pages

• Implementation:Tensorflow_Serving_servable_handle_h