Principle:Arize ai Phoenix Server Side Span Ingestion

Knowledge Sources	Phoenix gRPC Server Phoenix OTLP Decoder Phoenix HTTP Traces Router
Domains	AI Observability, Telemetry Ingestion, Data Persistence
Last Updated	2026-02-14 00:00 GMT

Overview

Server-side telemetry ingestion is the process by which a collector receives OTLP-encoded spans from client exporters, decodes them from protobuf into internal representations, extracts semantic attributes, and persists them to a database for query and visualization.

Description

While the client-side of the trace ingestion pipeline focuses on producing and transmitting spans, the server-side focuses on receiving, decoding, and storing them. In the Phoenix architecture, the server provides two parallel ingestion pathways:

HTTP Receiver

The HTTP receiver is implemented as a FastAPI route at POST /v1/traces. It accepts OTLP ExportTraceServiceRequest messages serialized as protobuf with Content-Type: application/x-protobuf. The receiver supports both gzip and deflate content encoding for compressed payloads. After parsing, spans are processed asynchronously via FastAPI's BackgroundTasks mechanism.

gRPC Receiver

The gRPC receiver implements the standard OTLP TraceService.Export RPC as defined in the OpenTelemetry protobuf specification. It operates as an async gRPC server (using grpc.aio) listening on a configurable port (default 4317). The gRPC server supports TLS with optional mutual TLS (mTLS) for client certificate verification, and API key authentication via a server interceptor.

Common Decoding Pipeline

Both receivers share the same span decoding logic:

1. Iterate resource_spans: Each ExportTraceServiceRequest contains one or more ResourceSpans, each representing spans from a single instrumented service instance.
2. Extract project name: The project name is extracted from the resource attributes (using the openinference.project.name key).
3. Iterate scope_spans and spans: Each ResourceSpans contains one or more ScopeSpans (grouped by instrumentation scope), each containing individual spans.
4. Decode OTLP span: The protobuf span is decoded into Phoenix's internal Span schema, converting trace/span IDs from binary to hex strings, timestamps from nanoseconds to datetime objects, attributes from nested key-value protobuf structures to Python dictionaries, and events (including exceptions) to typed event objects.
5. Enqueue for persistence: The decoded span is enqueued along with its project name for asynchronous database insertion.

Usage

Use this principle whenever:

• Understanding how traces flow from client applications to the Phoenix database.
• Diagnosing ingestion failures (HTTP 415/422/503 errors or gRPC status codes).
• Configuring server-side authentication for secured collector endpoints.
• Deciding between HTTP and gRPC receivers based on infrastructure requirements.
• Tuning ingestion capacity (the server has a span queue with a capacity limit and returns HTTP 503 when full).

Theoretical Basis

The ingestion pipeline follows a multi-stage processing model:

Client Exporter
       |
       | OTLP Protobuf (HTTP POST /v1/traces  OR  gRPC ExportTraceService)
       v
[Authentication Layer]   <-- API key interceptor / bearer auth
       |
       v
[Payload Decompression]   <-- gzip / deflate (HTTP only)
       |
       v
[Protobuf Deserialization]   <-- ExportTraceServiceRequest.ParseFromString()
       |
       v
[Span Decoding]   <-- decode_otlp_span() per span
       |
       | Converts: binary IDs -> hex strings
       | Converts: unix_nano timestamps -> datetime
       | Converts: protobuf KeyValue -> Python dict
       | Extracts: OpenInference span_kind
       | Decodes: status codes, events, exceptions
       v
[Span Queue]   <-- Async enqueue with project_name
       |
       v
[Database Persistence]   <-- SQLAlchemy insert into spans table

Protobuf to Internal Schema Mapping

OTLP Protobuf Field	Internal Span Field	Transformation
trace_id (bytes)	context.trace_id (str)	Hex encoding via hexlify()
span_id (bytes)	context.span_id (str)	Hex encoding via hexlify()
parent_span_id (bytes)	parent_id (Optional[str])	Hex encoding, None if empty
start_time_unix_nano (int)	start_time (datetime)	datetime.fromtimestamp(ns / 1e9, tz=utc)
end_time_unix_nano (int)	end_time (datetime)	datetime.fromtimestamp(ns / 1e9, tz=utc)
attributes (KeyValue[])	attributes (dict)	Recursive decoding, unflattening, JSON string loading
status (Status)	status_code, status_message	Enum mapping (UNSET, OK, ERROR)
events (Event[])	events (list[SpanEvent])	Exception events become SpanException instances
name (str)	name (str)	Direct mapping

Capacity Management

The HTTP receiver checks span queue capacity before accepting new requests. When the queue is full, it responds with HTTP 503 Service Unavailable and increments a Prometheus counter (SPAN_QUEUE_REJECTIONS). This back-pressure mechanism prevents the server from being overwhelmed by high-volume ingestion bursts.

Security Model

Both receivers support authentication:

• HTTP: Bearer token validation via FastAPI dependency injection.
• gRPC: ApiKeyInterceptor that validates tokens from the token_store before allowing the RPC to proceed.
• TLS: The gRPC server supports TLS with optional client certificate verification (mTLS) configured via environment variables.

Related Pages

Implemented By

• Implementation:Arize_ai_Phoenix_OTLP_Server_Receiver

Uses Heuristic

• Heuristic:Arize_ai_Phoenix_Queue_Clear_Race_Condition