Workflow:Tensorflow Serving REST API Inference
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Model_Serving, API_Design |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
End-to-end process for making inference requests to TensorFlow Serving models using the RESTful HTTP API, covering all API methods (Predict, Classify, Regress), JSON tensor encoding, and response handling.
Description
This workflow covers the complete REST API interaction pattern for TensorFlow Serving. The HTTP server exposes endpoints for model status queries, metadata retrieval, and three inference methods: Predict (generic tensor operations), Classify (returns class labels and scores), and Regress (returns numeric values). Requests and responses use JSON encoding, with support for row-format (instances) and columnar-format (inputs) tensor specification, binary data encoding via base64, and special floating point values.
Usage
Execute this workflow when building client applications that communicate with TensorFlow Serving via HTTP/REST rather than gRPC. This is common for web applications, microservice architectures, and environments where protobuf tooling is unavailable. The REST API is slightly less performant than gRPC but offers broader client compatibility.
Execution Steps
Step 1: Start Server with REST Endpoint
Launch TensorFlow Serving with the REST API port enabled. By default, Docker images expose port 8501 for REST. When running the binary directly, use the --rest_api_port flag. The server accepts JSON requests and returns JSON responses.
Key considerations:
- Docker images expose port 8501 for REST and port 8500 for gRPC by default
- Both ports can be active simultaneously
- Use --rest_api_timeout_in_ms to configure request timeout for the REST endpoint
- The REST API does not support the MultiInference endpoint (gRPC only)
Step 2: Query Model Status and Metadata
Before sending inference requests, verify the model is loaded and available by querying the model status endpoint. Optionally retrieve model metadata to discover available signatures, input/output tensor specifications, and version information.
Key considerations:
- Model status: GET /v1/models/{MODEL_NAME}
- Model metadata: GET /v1/models/{MODEL_NAME}/metadata
- Specific version: append /versions/{VERSION} or /labels/{LABEL} to the URL
- The status response includes version state (AVAILABLE, LOADING, UNLOADING)
Step 3: Format the Request Payload
Construct the JSON request body according to the target API method. For Predict requests, use either row format (instances key with a list of input objects) or columnar format (inputs key with named tensor arrays). For Classify and Regress, provide examples with feature name/value pairs.
Key considerations:
- Row format requires all named inputs to have the same 0th dimension (batch size)
- Columnar format allows inputs with different 0th dimensions
- Binary data (e.g., image bytes) must be base64-encoded using the {"b64": "..."} wrapper
- Optionally specify signature_name to target a non-default serving signature
Step 4: Send Inference Request
POST the JSON payload to the appropriate inference endpoint. The URL structure encodes the model name, optional version/label, and the API method. The server parses the JSON, converts values to TensorFlow tensors, runs inference through the model graph, and returns results as JSON.
Key considerations:
- Predict: POST /v1/models/{MODEL_NAME}:predict
- Classify: POST /v1/models/{MODEL_NAME}:classify
- Regress: POST /v1/models/{MODEL_NAME}:regress
- Content-Type should be application/json
- JSON supports NaN, Infinity, and -Infinity as special float tokens
Step 5: Parse the Response
Extract inference results from the JSON response. Predict responses contain predictions (row format) or outputs (columnar format). Classify responses contain lists of class label/score pairs. Regress responses contain a list of numeric values. Error responses include an error key with a message string.
Key considerations:
- Output tensor names ending in _bytes are automatically base64-encoded in the response
- Row-format requests produce predictions key; columnar-format produces outputs key
- HTTP status codes indicate error types (404 for model not found, 400 for bad request)
- Floating point precision may be limited by JSON number representation