Principle:Langfuse Langfuse Eval Template Variable Extraction

Knowledge Sources	Langfuse
Domains	LLM Evaluation, Data Extraction
Last Updated	2026-02-14 00:00 GMT

Overview

Variable Extraction is the principle of resolving template variable placeholders into concrete string values by looking up data from traces, observations, and dataset items using a configurable mapping schema and optional JSONPath selectors.

Description

Evaluation templates contain prompts with variable placeholders (e.g., {{input}}, {{output}}) that must be substituted with actual data from the trace being evaluated before the prompt can be sent to the LLM judge. Variable Extraction bridges the gap between the abstract evaluation template and the concrete trace data.

Each variable in a template is associated with a variable mapping that specifies:

1. The Langfuse object type -- Which data source to query: "trace" (the trace itself), an observation type (e.g., "generation", "span"), or "dataset_item" (a dataset item linked via a dataset run).

1. The column identifier -- Which column of the data source to extract from (e.g., "input", "output", "metadata", "name").

1. The object name -- For observations, the name of the specific observation within the trace to target (since a trace may contain multiple observations).

1. The JSON selector -- An optional JSONPath expression to extract a specific value from within a JSON column. This allows reaching into nested data structures such as extracting a specific key from a trace's metadata JSON.

The extraction process iterates through each template variable, resolves its mapping, fetches the corresponding data from either ClickHouse (for traces and observations) or PostgreSQL (for dataset items), applies any JSONPath selector, and converts the result to a string. The process also extracts the environment from trace/observation data, which is propagated to the score created by the evaluation.

Usage

Use Variable Extraction when:

• You need to understand how evaluation prompts get populated with real data
• You are debugging why an evaluation produced unexpected results (often caused by incorrect variable mappings)
• You are extending the set of available data sources or columns for evaluation variables
• You are building new variable mapping UIs or APIs

Theoretical Basis

The Variable Extraction principle implements a map-fetch-transform pipeline with internal caching:

Step 1 - Initialize Caches:

traceCache = Map<string, TraceDomain | null>()
observationCache = Map<string, Observation | null>()
results = []

Step 2 - Sequential Variable Resolution:

Variables are resolved sequentially (not in parallel) to maximize cache hit rates. When multiple variables map to the same trace or observation, only the first requires a database lookup.

FOR EACH variable IN template.vars:
  mapping = FIND mapping WHERE templateVariable == variable

  IF mapping NOT found:
    results.PUSH({ var: variable, value: "" })
    CONTINUE

  SWITCH mapping.langfuseObject:
    CASE "dataset_item":
      item = QUERY PostgreSQL dataset_items
        WHERE id = datasetItemId
        AND project_id = projectId
        AND (valid_from = version OR valid_to IS NULL)
      IF item NOT found: THROW Error
      value = EXTRACT_AND_TRANSFORM(item, mapping)
      results.PUSH({ var: variable, value })

    CASE "trace":
      trace = traceCache.GET(traceId)
        OR FETCH from ClickHouse AND CACHE
      IF trace NOT found: THROW Error
      value = EXTRACT_AND_TRANSFORM(trace, mapping)
      results.PUSH({ var: variable, value, environment: trace.environment })

    CASE observation_type (generation, span, event):
      observation = observationCache.GET(traceId + objectName)
        OR FETCH from ClickHouse by name AND CACHE
      IF observation NOT found: THROW UnrecoverableError
      value = EXTRACT_AND_TRANSFORM(observation, mapping)
      results.PUSH({ var: variable, value, environment: observation.environment })

RETURN results

Step 3 - Extract and Transform:

FUNCTION EXTRACT_AND_TRANSFORM(row, mapping):
  rawValue = row[mapping.selectedColumnId]

  IF mapping.jsonSelector:
    TRY:
      parsed = IF string THEN JSON.parse(rawValue) ELSE rawValue
      jsonResult = JSONPath(mapping.jsonSelector, parsed)
      RETURN toString(jsonResult)
    CATCH:
      RETURN toString(rawValue)  // Fallback to original value on JSONPath error
  ELSE:
    RETURN toString(rawValue)

Step 4 - Type Coercion:

All extracted values are converted to strings for prompt substitution:

• null/undefined produce an empty string
• Strings, numbers, and booleans are converted via .toString()
• Objects are serialized via JSON.stringify()
• All other types use String() coercion

Caching Strategy:

The function maintains per-invocation caches (not global caches) that live only for the duration of a single evaluation job. This is important because:

• Trace and observation data may change between evaluations
• The cache key for traces is projectId:traceId
• The cache key for observations is projectId:traceId:objectName
• Dataset items are not cached because PostgreSQL queries are cheaper than ClickHouse queries

Environment Propagation:

The environment string (e.g., "production", "staging") is extracted alongside variable values from trace and observation data. The first variable that yields an environment value is used as the environment for the resulting evaluation score. This ensures scores inherit the deployment context of the data they evaluate.

Related Pages

Implemented By

• Implementation:Langfuse_Langfuse_ExtractVariablesFromTracingData