Principle:Langfuse Langfuse Experiment Configuration Validation
| Knowledge Sources | |
|---|---|
| Domains | LLM Experimentation, Configuration Validation |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Experiment configuration validation is the practice of verifying that a prompt template and a dataset are structurally compatible before an experiment run is initiated, ensuring that every variable referenced in the prompt can be satisfied by at least one column present in the dataset items.
Description
When running prompt experiments at scale, a mismatch between the variables a prompt expects and the fields a dataset provides is one of the most common sources of wasted compute. An experiment that starts executing only to fail on every item because the dataset lacks a required column results in orphaned traces, unnecessary LLM API calls, and a confusing user experience.
Experiment configuration validation addresses this by performing a lightweight, pre-execution compatibility check. The validation procedure extracts all template variables (those delimited by double curly braces, e.g. {{variable}}) and message placeholders from the prompt, then inspects every active dataset item to determine whether its input fields can supply values for those variables. The result is a structured report that tells the caller:
- Whether the configuration is valid at all.
- How many total dataset items exist.
- A per-variable count showing how many items contain each variable.
This gives the user enough information to decide whether to proceed, adjust the prompt, or enrich the dataset before committing resources.
Usage
Use experiment configuration validation whenever:
- A user selects a prompt and dataset combination in the experiment creation UI and the system needs to provide immediate feedback.
- An automated pipeline needs to gate experiment execution behind a compatibility check.
- A dataset has been recently modified and the user wants to confirm it still matches the prompts that reference it.
- The experiment involves an optional dataset version filter, which could reduce the set of available items and change variable coverage.
Theoretical Basis
The validation algorithm operates in four sequential phases:
Phase 1 -- Prompt Variable Extraction
Given a prompt (text or chat type), the algorithm extracts two kinds of dynamic slots:
- Template variables: Identified by scanning the prompt string (or its JSON-serialized form for chat prompts) for the pattern
{{variableName}}. Each unique variable name is collected into a set. - Message placeholders: For chat-type prompts, an additional scan identifies named placeholder slots that accept arrays of messages rather than simple string substitution.
The union of template variables and message placeholders forms the complete set of required variables.
Phase 2 -- Dataset Item Retrieval
All active items for the specified dataset are fetched, optionally filtered by a dataset version timestamp. Items that are archived or inactive are excluded to reflect what an actual experiment run would process.
Phase 3 -- Variable Matching
Each dataset item's input is inspected against the required variable set:
- If the input is a string and there is exactly one variable, the item is counted as satisfying that variable.
- If the input is a JSON object, each key is compared against the variable set, and matching keys increment the corresponding variable count.
- Items whose input is null, missing, or otherwise invalid are skipped entirely.
The output of this phase is a map from variable name to the count of dataset items that contain it.
Phase 4 -- Validity Decision
The configuration is deemed invalid if any of the following conditions hold:
- The prompt contains no variables or placeholders at all (there is nothing to substitute).
- The dataset contains zero active items (or zero items at the specified version).
- No dataset item contains any of the required variables (the variable map is empty).
Otherwise, the configuration is valid, and the response includes the total item count and the variable map.
FUNCTION validateExperimentConfig(prompt, dataset, version?):
variables = extractTemplateVariables(prompt) UNION extractPlaceholders(prompt)
IF variables is empty:
RETURN { isValid: false, message: "Prompt has no variables" }
items = fetchActiveDatasetItems(dataset, version)
IF items is empty:
RETURN { isValid: false, message: "Dataset is empty" }
variablesMap = {}
FOR EACH item IN items:
IF item.input is valid AND matches variables:
FOR EACH var IN variables:
IF var IN item.input:
variablesMap[var] += 1
IF variablesMap is empty:
RETURN { isValid: false, message: "No items contain any variables" }
RETURN { isValid: true, totalItems: |items|, variablesMap }