Implementation:Tensorflow Serving Resource Tracker
| Knowledge Sources | |
|---|---|
| Domains | Resource Management, Serving |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
A resource availability tracker that maintains total and used resource allocations, determining whether sufficient resources exist to load new servables.
Description
ResourceTracker keeps track of total system resources and currently used resources in a serving system. It is created via a static Create() factory method that validates and normalizes the total_resources (which must be bound to specific device instances) and takes ownership of a ResourceUtil for performing resource arithmetic. The key method ReserveResources() estimates a servable's resource needs via its Loader, conservatively overbinds the current used resources (to handle unbound allocations), adds the servable's requirements, and checks whether the result fits within total resources using LessThanOrEqual. If it fits, the servable's resources are added to used_resources and success is set to true. RecomputeUsedResources() recalculates used resources from scratch given a list of all active loaders (loaded, loading, or unloading servables). The class is not thread-safe and must be externally synchronized.
Usage
Use this in the serving model manager to make admission control decisions: before approving a servable for loading, call ReserveResources to check if the system has enough capacity. Periodically call RecomputeUsedResources to reconcile the tracked state with reality.
Code Reference
Source Location
- Repository: Tensorflow_Serving
- File:
tensorflow_serving/resources/resource_tracker.h(header),tensorflow_serving/resources/resource_tracker.cc(implementation) - Lines: 1-83 (header), 1-92 (implementation)
Signature
class ResourceTracker {
public:
static Status Create(const ResourceAllocation& total_resources,
std::unique_ptr<ResourceUtil> util,
std::unique_ptr<ResourceTracker>* tracker);
~ResourceTracker() = default;
Status ReserveResources(const Loader& servable, bool* success);
Status RecomputeUsedResources(const std::vector<const Loader*>& servables);
const ResourceAllocation& total_resources() const;
const ResourceAllocation& used_resources() const;
};
Import
#include "tensorflow_serving/resources/resource_tracker.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| total_resources | ResourceAllocation |
Yes (Create) | The total system resources; must be valid and bound |
| util | std::unique_ptr<ResourceUtil> |
Yes (Create) | Utility for resource arithmetic operations |
| servable | const Loader& |
Yes (ReserveResources) | The loader whose resource needs are being checked |
| servables | std::vector<const Loader*> |
Yes (Recompute) | All active loaders for recomputing used resources |
Outputs
| Name | Type | Description |
|---|---|---|
| Status | tensorflow::Status |
OK on success, error on invalid data |
| success | bool* |
Whether the reservation succeeded (output parameter of ReserveResources) |
| total_resources() | const ResourceAllocation& |
The total resources configured for the system |
| used_resources() | const ResourceAllocation& |
The currently used/reserved resources |
Usage Examples
Checking Resource Availability
// Create the tracker
ResourceUtil::Options options;
options.devices = {{"CPU", 1}, {"GPU", 2}};
auto util = std::make_unique<ResourceUtil>(options);
std::unique_ptr<ResourceTracker> tracker;
TF_RETURN_IF_ERROR(ResourceTracker::Create(total_resources, std::move(util), &tracker));
// Attempt to reserve resources for a new servable
bool success;
TF_RETURN_IF_ERROR(tracker->ReserveResources(*loader, &success));
if (success) {
// Proceed with loading the servable
}