Implementation:Langfuse Langfuse ExperimentsRouter ValidateConfig
| Knowledge Sources | |
|---|---|
| Domains | LLM Experimentation, Configuration Validation |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Concrete tool for validating experiment configuration compatibility between a prompt and a dataset, provided by Langfuse.
Description
experimentsRouter.validateConfig is a tRPC query procedure exposed by the Langfuse web application. It accepts a project ID, dataset ID, prompt ID, and an optional dataset version, then performs a multi-step compatibility check to determine whether the prompt's template variables and message placeholders can be satisfied by the dataset's item inputs.
The procedure operates entirely within the web server (no worker involvement). It first resolves the prompt via PromptService, which handles prompt inheritance and composition. It then extracts template variables using extractVariables (which finds {{var}} patterns) and placeholder names using extractPlaceholderNames (for chat-type prompts with message placeholder slots). The union of these two sets forms the complete variable requirement.
Active dataset items are retrieved from ClickHouse via getDatasetItems, filtered by dataset ID and optionally by version. Each item's input is validated with validateDatasetItem and then counted using countValidDatasetItems, which produces a Record<string, number> mapping each variable name to the number of items that contain it.
The response uses a Zod discriminated union: on success (isValid: true), it includes totalItems and variablesMap; on failure (isValid: false), it includes a human-readable message describing the problem.
Usage
This procedure is called from the experiment creation UI when a user selects a prompt and dataset. The frontend invokes trpc.experiments.validateConfig.useQuery() to provide real-time feedback before the user clicks "Run Experiment". It is also useful for any programmatic pre-flight check.
Code Reference
Source Location
- Repository: langfuse
- File: web/src/features/experiments/server/router.ts
- Lines: 80-175
Signature
experimentsRouter.validateConfig: protectedProjectProcedure
.input(
z.object({
projectId: z.string(),
datasetId: z.string(),
promptId: z.string(),
datasetVersion: z.coerce.date().optional(),
}),
)
.output(ConfigResponse)
.query(async ({ input, ctx }) => { ... })
Import
import { experimentsRouter } from "@/src/features/experiments/server/router";
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| projectId | string | Yes | The project to which the prompt and dataset belong. Used for authorization and scoping. |
| datasetId | string | Yes | The ID of the dataset whose items will be checked for variable coverage. |
| promptId | string | Yes | The ID of the prompt from which template variables and placeholders are extracted. |
| datasetVersion | Date (coerced) | No | Optional version timestamp to filter dataset items. Only items valid at this version are considered. |
Outputs
| Name | Type | Description |
|---|---|---|
| isValid | boolean (literal true or false) | Discriminant field indicating whether the configuration passed validation. |
| totalItems | number | (Only when isValid is true) The total count of active dataset items found. |
| variablesMap | Record<string, number> | (Only when isValid is true) A map from each prompt variable name to the number of dataset items that contain it. |
| message | string | (Only when isValid is false) A human-readable explanation of why validation failed. |
Usage Examples
Basic Validation Query
const result = await trpc.experiments.validateConfig.query({
projectId: "proj_abc123",
datasetId: "ds_def456",
promptId: "prompt_ghi789",
});
if (result.isValid) {
console.log(`Ready to run: ${result.totalItems} items available`);
console.log("Variable coverage:", result.variablesMap);
// e.g. { question: 42, context: 40 }
} else {
console.error(`Validation failed: ${result.message}`);
}
Validation with Dataset Version Filter
const result = await trpc.experiments.validateConfig.query({
projectId: "proj_abc123",
datasetId: "ds_def456",
promptId: "prompt_ghi789",
datasetVersion: new Date("2025-12-01T00:00:00Z"),
});
if (!result.isValid) {
// Possible messages:
// - "Selected prompt not found."
// - "Selected prompt has no variables or placeholders."
// - "Selected dataset is empty or all items are inactive."
// - "No dataset item contains any variables."
alert(result.message);
}