Heuristic:Langfuse Langfuse ClickHouse FINAL Skip Optimization
| Knowledge Sources | |
|---|---|
| Domains | Performance, ClickHouse |
| Last Updated | 2026-02-14 06:00 GMT |
Overview
Skip the expensive ClickHouse FINAL modifier for OpenTelemetry projects because OTel spans are immutable, eliminating the need for deduplication and significantly improving query performance.
Description
ClickHouse uses ReplacingMergeTree tables that can contain duplicate rows until background merges complete. The FINAL modifier forces deduplication at query time, but this is expensive (full table scan per partition). OpenTelemetry spans are immutable by design: once a span is completed, it is never updated. This means OTel data never has duplicates that need deduplication, making FINAL unnecessary and wasteful for those projects.
Usage
This heuristic is automatically applied when LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS is set to true. The system checks Redis to determine if a project uses OTel ingestion (tracked via 24-hour TTL keys) and skips FINAL accordingly. For non-OTel projects, FINAL can be globally disabled via LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL.
The Insight (Rule of Thumb)
- Action: Skip the
FINALmodifier on ClickHouse queries for OTel-ingesting projects. - Value: Significant query performance improvement (avoids full partition scan for deduplication).
- Trade-off: None for OTel projects (spans are immutable). Non-OTel projects still need
FINALorLIMIT 1 BYfor correctness. - Feature Flags:
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS(per-project),LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL(global override).
Reasoning
The multi-layer decision logic prioritizes correctness:
- Check
LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL(global override, takes precedence) - If
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTSis enabled, check Redis for project OTel usage - Default to using
FINAL(safe default)
OTel project tracking uses Redis keys with 24-hour TTL, set during OTel ingestion. This means the optimization activates automatically after a project's first OTel ingestion event.
// From packages/shared/src/server/queries/clickhouse-sql/query-options.ts
export async function shouldSkipObservationsFinal(projectId: string): Promise<boolean> {
// 1. Global override
if (env.LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL === "true") return true;
// 2. Per-project OTel check
if (env.LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS === "true") {
return await isOtelProject(projectId); // Checks Redis TTL key
}
return false;
}