Heuristic:Langfuse Langfuse Eval Loop Prevention
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Architecture |
| Last Updated | 2026-02-14 06:00 GMT |
Overview
Internal traces generated by evaluation and experiment LLM calls are tagged with a langfuse-* environment prefix to prevent infinite evaluation loops where an eval trace triggers another eval.
Description
When Langfuse runs an LLM evaluation, it creates a trace of that LLM call (for observability). This trace, if not specially marked, would trigger the evaluation pipeline again (since new traces trigger evaluations), creating an infinite loop: user trace -> eval -> eval trace -> eval -> eval trace -> ... The solution is to tag all internal traces with an environment starting with langfuse (using the LangfuseInternalTraceEnvironment enum) and skip eval job creation for traces matching this prefix.
Usage
This heuristic is automatically enforced in two locations that must stay synchronized:
fetchLLMCompletion.ts: Sets thelangfuse-*environment on internal traces.evalService.ts(createEvalJobs): Filters out traces withlangfuse-*environments from eval triggering.
If you add new internal LLM call paths, you must ensure they use the LangfuseInternalTraceEnvironment enum.
The Insight (Rule of Thumb)
- Action: Tag all internal LLM traces with a
langfuse-*environment prefix and filter them out of eval job creation. - Value: Prevents infinite recursion in the eval pipeline.
- Trade-off: Internal traces are still stored and visible but do not trigger evaluations.
- Coordination: The tag is set in
fetchLLMCompletion.tsand checked inevalService.ts. Both files must use the sameLangfuseInternalTraceEnvironmentenum.
Reasoning
The evaluation pipeline listens for trace upsert events and creates eval jobs for matching configurations. Without the environment filter, the pipeline would enter an infinite recursion:
- User creates a trace
- Trace upsert triggers eval job creation
- Eval job runs an LLM call, which creates a new trace
- New trace triggers another eval job creation
- Infinite loop
The source code explicitly documents this with a cross-reference between the two files:
// From packages/shared/src/server/llm/fetchLLMCompletion.ts
// This prevents infinite eval loops (user trace -> eval -> eval trace -> another eval)
// See corresponding check in worker/src/features/evaluation/evalService.ts createEvalJobs()
traceSinkParams: {
environment: LangfuseInternalTraceEnvironment.Evaluation,
}
// From worker/src/features/evaluation/evalService.ts
// Skip internal traces to prevent infinite eval loops
if (trace.environment?.startsWith("langfuse")) {
return; // Don't create eval jobs for internal traces
}