Implementation:Huggingface Datatrove EndpointInferenceServer

Knowledge Sources	Huggingface_Datatrove
Domains	Machine Learning Inference, API Integration
Last Updated	2026-02-14 17:00 GMT

Overview

EndpointServer is an inference server implementation that sends requests to an external endpoint URL using an OpenAI-compatible API, supporting both chat and completion modes.

Description

The EndpointServer class extends InferenceServer to provide connectivity to remote or local inference endpoints. It distinguishes between localhost HTTP endpoints and external HTTPS endpoints during initialization. For external endpoints, it creates an AsyncOpenAI client with the configured API key, base URL, and request timeout. For localhost HTTP endpoints, it falls back to the base class HTTP implementation, avoiding the OpenAI dependency entirely.

The server implements the full request lifecycle by overriding _make_request to dispatch inference calls through the OpenAI client library. It handles both chat completions (via client.chat.completions.create) and standard completions (via client.completions.create), normalizing the response into a consistent dictionary format containing choices and usage statistics. The method also implements a comprehensive error handling strategy that classifies OpenAI exceptions into retryable errors (connection, timeout, rate limit, internal server) and non-retryable errors (authentication, bad request, not found, permission denied).

Since EndpointServer connects to an already-running external service, its start_server method is a no-op, and its monitor_health method awaits an indefinitely pending future to satisfy the abstract interface. The _wait_until_ready method uses shorter delays compared to the base class since it only needs to verify endpoint availability rather than wait for a local server to start.

Usage

Use EndpointServer when you need to run inference against a pre-existing remote API endpoint such as a Hugging Face Inference Endpoint, OpenAI API, or any OpenAI-compatible server. It is particularly useful for distributed processing where the inference service is hosted externally rather than managed locally.

Code Reference

Source Location

• Repository: Huggingface_Datatrove
• File: src/datatrove/pipeline/inference/servers/endpoint_server.py
• Lines: 1-151

Signature

class EndpointServer(InferenceServer):
    def __init__(self, config: "InferenceConfig", rank: int):
        ...

    async def start_server(self) -> None:
        ...

    async def monitor_health(self) -> None:
        ...

    async def _wait_until_ready(self, max_attempts: int = 10, delay_sec: float = 1.0) -> None:
        ...

    def get_base_url(self) -> str:
        ...

    async def _make_request(self, payload: dict) -> dict:
        ...

Import

from datatrove.pipeline.inference.servers.endpoint_server import EndpointServer

I/O Contract

Inputs

Name	Type	Required	Description
config	InferenceConfig	Yes	Configuration object containing endpoint_url, api_key, request_timeout, use_chat, and model settings
rank	int	Yes	Rank of the server instance for distributed execution
payload	dict	Yes (for _make_request)	Request payload dictionary containing model name and generation parameters

Outputs

Name	Type	Description
response	dict	Parsed JSON response dict with "choices" (containing text/message and finish_reason) and "usage" (prompt_tokens, completion_tokens, total_tokens)

Usage Examples

Basic Usage

from datatrove.pipeline.inference.servers.endpoint_server import EndpointServer

# Assuming an InferenceConfig with endpoint_url set
server = EndpointServer(config=inference_config, rank=0)

# The server connects to the external endpoint
# No local server startup is required
await server._wait_until_ready()

# Make a chat completion request
response = await server._make_request({
    "model": "my-model",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100,
})

Related Pages

• Principle:Huggingface_Datatrove_Inference_Server_Management