Implementation:Tensorflow Serving CreateAndStartHttpServer

Knowledge Sources	TensorFlow Serving
Domains	Infrastructure, Networking
Last Updated	2026-02-13 17:00 GMT

Overview

Concrete tool for creating and starting the embedded HTTP server for REST API inference provided by the TensorFlow Serving http_server module.

Description

CreateAndStartHttpServer() is a factory function that:

1. Creates a ThreadPoolExecutor with the specified number of threads
2. Builds an HTTPServer using the net_http library
3. Creates a RestApiRequestDispatcher that matches URIs against (?i)/v1/.*
4. Optionally sets up a Prometheus metrics endpoint at /monitoring/prometheus/metrics
5. Starts the server in accepting mode and returns the server handle

The dispatcher creates an HttpRestApiHandler and delegates all matched requests to its ProcessRequest() method.

Usage

Called by Server::BuildAndStart() when http_port > 0. Not called directly by users; configured via --rest_api_port CLI flag.

Code Reference

Source Location

• Repository: tensorflow/serving
• File: tensorflow_serving/model_servers/http_server.cc
• Lines: L212-253 (implementation)
• Header: tensorflow_serving/model_servers/http_server.h L33-35

Signature

std::unique_ptr<net_http::HTTPServerInterface> CreateAndStartHttpServer(
    int port,                                    // TCP port to listen on
    int num_threads,                             // HTTP thread pool size
    int timeout_in_ms,                           // Request timeout (default 30000)
    const MonitoringConfig& monitoring_config,   // Prometheus config
    ServerCore* core                             // Model server core (not owned)
);

Import

#include "tensorflow_serving/model_servers/http_server.h"

I/O Contract

Inputs

Name	Type	Required	Description
port	int	Yes	TCP port for HTTP server
num_threads	int	Yes	Thread pool size (default 4*NumCPUs)
timeout_in_ms	int	Yes	Request timeout in milliseconds
monitoring_config	MonitoringConfig	Yes	Prometheus monitoring configuration
core	ServerCore*	Yes	Pointer to loaded ServerCore with models

Outputs

Name	Type	Description
HTTPServerInterface	unique_ptr	Running HTTP server accepting REST requests at /v1/models/...

Usage Examples

Enable REST API (CLI)

# Start server with REST API on port 8501
tensorflow_model_server \
    --port=8500 \
    --rest_api_port=8501 \
    --model_name=mnist \
    --model_base_path=/tmp/mnist_model

Docker with REST API

docker run -p 8501:8501 \
    --mount type=bind,source=/path/to/model,target=/models/my_model \
    -e MODEL_NAME=my_model \
    -t tensorflow/serving

Related Pages

Implements Principle

• Principle:Tensorflow_Serving_HTTP_Server_Setup