Principle:Tensorflow Serving Source Routing
| Knowledge Sources | |
|---|---|
| Domains | Model Serving, Core Framework |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
The Source Routing principle defines how aspired-version calls from a single source are demultiplexed to multiple downstream targets based on configurable routing criteria.
Description
In a multi-model serving environment, a single Source may produce aspired-version notifications for different types of servables (e.g., different model architectures). These need to be routed to different processing pipelines (SourceAdapters) that know how to handle each type. Source Routing provides this demultiplexing capability.
The routing architecture consists of:
- SourceRouter<T> - The abstract base class that acts as both a Target (receiving input) and a Source (providing N output ports). Each output port is an IdentitySourceAdapter that passes through data unchanged.
- DynamicSourceRouter<T> - Routes based on an explicit name-to-port map that can be updated at runtime. The last port is reserved as the default for unmatched names.
- StaticSourceRouter<T> - Routes based on cascading substring matching against the servable name. Configuration is immutable after creation.
The routing decision is made per aspired-version call by the subclass-implemented Route() method. A special kNoRoute return value silently discards the call.
All routers lazily create their output ports on the first call to GetOutputPorts(), and SetAspiredVersions() blocks until ports are initialized.
Usage
Apply source routing when a single source produces servable notifications for multiple servable types that require different loading pipelines. Choose StaticSourceRouter for fixed routing based on naming conventions, or DynamicSourceRouter when the routing configuration may change at runtime.
Theoretical Basis
Source routing implements a demultiplexer pattern in the serving pipeline graph:
+-> Output Port 0 -> SourceAdapter A -> Manager
Source -> SourceRouter ---+-> Output Port 1 -> SourceAdapter B -> Manager
+-> Output Port N-1 (default) -> ...
Routing function:
Route(servable_name, versions) -> port_number | kNoRoute
Static routing: Cascading substring match
for i in 0..N-2:
if substring[i] in servable_name: return i
return N-1 (default)
Dynamic routing: Explicit map lookup
if servable_name in route_map: return route_map[servable_name]
return N-1 (default)
Key design properties:
- Single-input, multi-output: The router acts as a 1-to-N splitter in the pipeline graph.
- Lazy initialization: Output ports are created on first access, allowing subclasses to compute the port count dynamically.
- Thread-safe routing: DynamicSourceRouter protects its route map with a mutex; StaticSourceRouter needs no synchronization (immutable config).
- Fall-through default: Both implementations reserve the last port as a catch-all, ensuring no aspired-version calls are accidentally dropped.