Principle:Langfuse Langfuse OTel Endpoint Reception

Knowledge Sources	Langfuse
Domains	Observability, Ingestion, OpenTelemetry
Last Updated	2026-02-14 00:00 GMT

Overview

OTel Endpoint Reception is the principle of accepting OpenTelemetry trace data over HTTP, supporting multiple serialization formats and compression schemes, as the first stage of an ingestion pipeline.

Description

Modern observability platforms must interoperate with the OpenTelemetry ecosystem, which defines a standard wire protocol (OTLP) for exporting trace, metric, and log data. The OTel Endpoint Reception principle addresses the challenge of reliably accepting ExportTraceServiceRequest payloads from heterogeneous SDK clients -- including Python, JavaScript, Go, and others -- each of which may use different serialization formats (Protobuf binary or JSON) and compression (gzip or uncompressed).

The endpoint must perform several responsibilities atomically within a single HTTP request-response cycle:

1. Authentication and authorization -- validate that the caller holds a valid project API key and that ingestion has not been suspended for the project (e.g., due to usage threshold violations).
2. Body streaming and decompression -- since Next.js body parsing is disabled (via bodyParser: false), the raw request body is collected as a streaming buffer and optionally decompressed via gzip when the Content-Encoding: gzip header is present.
3. Content-type negotiation -- the endpoint inspects the Content-Type header to decide between Protobuf decoding (via a generated protobuf root) and JSON parsing. Strict exact matching is intentionally avoided to tolerate charsets and other suffixes.
4. Payload extraction -- the resourceSpans array is extracted from the parsed request body and validated for non-emptiness before forwarding downstream.
5. Header propagation -- configurable request headers are captured and forwarded for enterprise features such as ingestion masking.

The principle intentionally decouples reception from processing: once the resourceSpans are extracted, they are handed off to an asynchronous processing pipeline, allowing the HTTP response to return immediately.

Usage

Apply this principle whenever building an ingestion endpoint that must:

• Accept OTLP trace export requests from diverse OpenTelemetry SDK clients.
• Support both Protobuf and JSON serialization without requiring separate routes.
• Handle gzip-compressed payloads transparently.
• Enforce rate limiting and authentication at the API boundary.
• Minimize request latency by deferring heavy processing to a background queue.

Theoretical Basis

The OTel Endpoint Reception mechanism follows a staged pipeline pattern:

RECEIVE HTTP POST
    |
    v
AUTHENTICATE (project API key, check ingestion suspension)
    |
    v
STREAM RAW BODY (collect chunks into Buffer)
    |
    v
DECOMPRESS? (if Content-Encoding includes "gzip", apply gunzip)
    |
    v
NEGOTIATE FORMAT
    |--- Content-Type includes "application/x-protobuf"
    |       -> Decode via ExportTraceServiceRequest.decode()
    |       -> Convert to JS object via .toObject()
    |       -> Extract .resourceSpans
    |
    |--- Content-Type includes "application/json"
    |       -> JSON.parse(buffer.toString())
    |       -> Extract .resourceSpans
    |
    |--- Otherwise -> Return 400 Invalid content type
    |
    v
VALIDATE (resourceSpans is non-empty array)
    |
    v
PROPAGATE HEADERS (capture configurable header names for downstream use)
    |
    v
HAND OFF to OtelIngestionProcessor.publishToOtelIngestionQueue()
    |
    v
RETURN HTTP 200 (empty success response)

Key design decisions:

• Fail-fast on parse errors: Each parsing stage (body read, decompression, format decode) returns a 400 status immediately on failure, preventing malformed data from entering the pipeline.
• Empty-batch short-circuit: If resourceSpans is empty or absent, the endpoint returns an empty success response without enqueuing work.
• Rate limiting: The endpoint is wrapped with the ingestion rate limit resource, throttling excessive traffic per project before any processing occurs.
• Project flagging: Upon successful reception, the project is asynchronously marked as an OTel user via markProjectAsOtelUser(), enabling analytics on OTel adoption.

Related Pages

Implemented By

• Implementation:Langfuse_Langfuse_OTel_Traces_Handler