Principle:Langfuse Langfuse Post Ingestion Side Effects

Knowledge Sources	Langfuse
Domains	Event-Driven Architecture, Trace Ingestion
Last Updated	2026-02-14 00:00 GMT

Overview

Post-ingestion side effects are asynchronous reactions triggered after a trace is successfully written to the analytics database, enabling downstream systems like evaluations, webhooks, and change notifications to respond to new or updated trace data without blocking the ingestion pipeline.

Description

The Langfuse trace ingestion pipeline does not end when events are written to ClickHouse. Several downstream systems need to react to trace changes:

1. Evaluation job creation: When a trace is created or updated, evaluation rules configured for the project must be checked. If the trace matches an evaluation rule's filter criteria, an evaluation job is created and enqueued to the CreateEvalQueue. This powers the automated evaluation pipeline where LLM-as-judge or custom evaluators score traces.

1. Entity change notifications: The EntityChangeQueue receives notifications about trace changes, which can be used to invalidate caches, update real-time dashboards, or trigger integration sync.

1. Webhook dispatch: Projects can configure webhooks that fire when traces are created or updated. The WebhookQueue handles delivering these HTTP callbacks to external systems.

These side effects are decoupled from the main ingestion path through a dedicated TraceUpsertQueue. The key design decisions are:

Delayed Processing

The TraceUpsertQueue applies a 30-second default delay before jobs become eligible for processing. This delay serves a critical purpose: it allows the ingestion pipeline time to write all related events (trace, observations, scores) for a single user interaction before triggering side effects. Without this delay, an evaluation might run on a partial trace that is missing observations still in the queue, producing incorrect or incomplete results.

Sharding

Like the ingestion queue, the trace upsert queue supports sharding via LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT. The sharding key is typically {projectId}-{traceId}, ensuring that multiple upsert events for the same trace are routed to the same shard. This prevents race conditions where two workers simultaneously process side effects for the same trace.

Idempotency

Because the same trace may be updated multiple times in quick succession (e.g., as observations complete), and the delay window may cause multiple upsert events to be processed close together, all downstream side effect handlers must be idempotent. Running the same side effect twice for the same trace version should produce the same result.

Fan-Out Pattern

The TraceUpsertQueue acts as a fan-out point: a single trace upsert event triggers jobs on multiple downstream queues (CreateEvalQueue, EntityChangeQueue, WebhookQueue). This fan-out is handled by the worker that consumes TraceUpsertQueue jobs, which inspects the project's configuration to determine which downstream queues should receive jobs.

Usage

Apply this principle whenever:

• Adding a new system that needs to react to trace changes (e.g., a new integration, a notification system, a data export pipeline).
• Configuring the delay between trace ingestion and side effect execution.
• Designing idempotent handlers for trace-triggered workflows.
• Scaling the post-ingestion processing by adjusting shard counts.

Theoretical Basis

Event Flow

[IngestionQueue Worker]
    |
    |  Writes trace/observation/score to ClickHouse
    |
    v
[TraceUpsertQueue]  <-- 30-second delay by default
    |
    |  Worker picks up the job after delay
    |
    +-------+--------+--------+
    |       |        |        |
    v       v        v        v
[CreateEvalQueue]  [EntityChangeQueue]  [WebhookQueue]  [Other queues...]
    |               |                    |
    v               v                    v
  Run evaluations  Notify caches/UI    Deliver HTTP webhooks

Queue Configuration

TraceUpsertQueue default job options:
    removeOnComplete: 100     -- Keep last 100 completed jobs for debugging
    removeOnFail:     100,000 -- Keep failed jobs for investigation
    attempts:         Configurable via LANGFUSE_TRACE_UPSERT_QUEUE_ATTEMPTS
    delay:            30,000 ms (30 seconds)
    backoff:          Exponential, starting at 5,000 ms

Delay Justification

Timeline of a typical LLM application trace:

T+0ms    : trace-create event arrives
T+50ms   : span-create for retrieval step
T+200ms  : generation-create for LLM call
T+2000ms : generation-update with completion tokens
T+2100ms : span-update with end time
T+5000ms : score-create from inline evaluation

Without delay: TraceUpsertQueue processes at T+0ms
    -> Evaluation runs on empty trace (no observations)
    -> Webhook fires with incomplete data

With 30s delay: TraceUpsertQueue processes at T+30000ms
    -> All observations and scores have been written
    -> Evaluation runs on complete trace
    -> Webhook fires with full trace data

Sharding Consistency

Sharding key: "{projectId}-{traceId}"

This ensures that if two upsert events for the same trace arrive:
  1. Both are routed to the same shard.
  2. BullMQ processes them sequentially on that shard.
  3. Side effects are not duplicated by concurrent workers.

Shard count is controlled by:
  LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT

Queue name format:
  Shard 0: "trace-upsert-queue"
  Shard N: "trace-upsert-queue-{N}" (for N > 0)

Related Pages

Implemented By

• Implementation:Langfuse_Langfuse_TraceUpsertQueue