Implementation:Vibrantlabsai Ragas PydanticPrompt

Knowledge Sources	Vibrantlabsai_Ragas
Domains	LLM Evaluation, Prompt Engineering, Structured Output
Last Updated	2026-02-12 00:00 GMT

Overview

PydanticPrompt is a generic, type-safe prompt class that uses Pydantic models to define structured input and output contracts for LLM interactions, enabling validated generation with automatic output parsing and retry logic.

Description

PydanticPrompt is the primary prompt abstraction in the Ragas framework for producing structured, validated LLM outputs. It extends BasePrompt and is generic over two Pydantic model type parameters: InputModel and OutputModel. This design enforces a strict contract between the data fed into the prompt and the data expected back from the LLM.

The class automatically constructs a prompt string from three components: an instruction (natural language directive), an output signature (JSON Schema specification derived from the output Pydantic model), and optional few-shot examples (pairs of InputModel and OutputModel instances). When generate() or generate_multiple() is called, the prompt is formatted with the input data, sent to the LLM, and the raw text response is parsed back into a validated OutputModel instance using RagasOutputParser.

Key features include:

Type-safe I/O: Both inputs and outputs are Pydantic models, enabling compile-time type hints and runtime validation.
Automatic retry and output fixing: If the LLM returns malformed JSON, the RagasOutputParser invokes a secondary FixOutputFormat prompt to attempt repair, with configurable retry counts.
Multi-backend LLM support: The class dispatches to LangChain LLMs (via agenerate_prompt), Ragas LLMs (via generate), or InstructorLLM (via generate/agenerate with a response model).
Language adaptation: The adapt() method translates examples and optionally the instruction to a target language using a built-in TranslateStatements prompt.
Serialization: Prompts can be saved to and loaded from JSON files with version tracking and hash integrity checks.
Extensibility: Subclasses can override process_input() and process_output() hooks to transform data before prompt rendering or after parsing.

The module also defines several supporting classes: RagasOutputParser (a LangChain-compatible parser that wraps Pydantic validation with retry logic), FixOutputFormat (a self-referential PydanticPrompt for repairing bad output), and TranslateStatements (a PydanticPrompt for multilingual adaptation).

Usage

Import PydanticPrompt when you need to create a prompt that guarantees structured, Pydantic-validated outputs from an LLM. This is the standard approach for all Ragas evaluation metrics and any custom metric that requires deterministic output schemas. Use it instead of Prompt (simple prompt) when you need type-safe input/output contracts, automatic JSON parsing, or few-shot examples defined as Pydantic model instances.

Code Reference

Source Location

Repository: Vibrantlabsai_Ragas
File: src/ragas/prompt/pydantic_prompt.py

Signature

class PydanticPrompt(BasePrompt, t.Generic[InputModel, OutputModel]):
    input_model: t.Type[InputModel]
    output_model: t.Type[OutputModel]
    instruction: str
    examples: t.List[t.Tuple[InputModel, OutputModel]] = []

    def _generate_instruction(self) -> str: ...
    def _generate_output_signature(self, indent: int = 4) -> str: ...
    def _generate_examples(self) -> str: ...
    def to_string(self, data: t.Optional[InputModel] = None) -> str: ...

    async def generate(
        self,
        llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel],
        data: InputModel,
        temperature: t.Optional[float] = None,
        stop: t.Optional[t.List[str]] = None,
        callbacks: t.Optional[Callbacks] = None,
        retries_left: int = 3,
    ) -> OutputModel: ...

    async def generate_multiple(
        self,
        llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel],
        data: InputModel,
        n: int = 1,
        temperature: t.Optional[float] = None,
        stop: t.Optional[t.List[str]] = None,
        callbacks: t.Optional[Callbacks] = None,
        retries_left: int = 3,
    ) -> t.List[OutputModel]: ...

    def process_input(self, input: InputModel) -> InputModel: ...
    def process_output(self, output: OutputModel, input: InputModel) -> OutputModel: ...

    async def adapt(
        self,
        target_language: str,
        llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM],
        adapt_instruction: bool = False,
    ) -> "PydanticPrompt[InputModel, OutputModel]": ...

    def save(self, file_path: str) -> None: ...

    @classmethod
    def load(cls, file_path: str) -> "PydanticPrompt[InputModel, OutputModel]": ...

Import

from ragas.prompt import PydanticPrompt

I/O Contract

Inputs

Name	Type	Required	Description
input_model	t.Type[InputModel] (Pydantic BaseModel subclass)	Yes	The Pydantic model class that defines the shape of the input data
output_model	t.Type[OutputModel] (Pydantic BaseModel subclass)	Yes	The Pydantic model class that defines the expected output schema
instruction	str	Yes	Natural language instruction describing the task the LLM should perform
examples	List[Tuple[InputModel, OutputModel]]	No	Optional list of input/output example pairs for few-shot prompting (defaults to empty list)

generate() Parameters

Name	Type	Required	Description
llm	BaseRagasLLM or InstructorBaseRagasLLM or BaseLanguageModel	Yes	The language model to use for generation
data	InputModel	Yes	The input data instance conforming to the input_model schema
temperature	Optional[float]	No	Temperature parameter controlling randomness in generation
stop	Optional[List[str]]	No	Stop sequences to end generation
callbacks	Optional[Callbacks]	No	Callback functions invoked during the generation process
retries_left	int	No	Number of retry attempts for invalid LLM responses (default: 3)

Outputs

Name	Type	Description
return (generate)	OutputModel	A single validated Pydantic model instance parsed from the LLM response
return (generate_multiple)	List[OutputModel]	A list of validated Pydantic model instances parsed from the LLM responses
return (adapt)	PydanticPrompt[InputModel, OutputModel]	A new prompt instance with translated examples and optionally translated instruction

Usage Examples

Defining a Custom PydanticPrompt

from pydantic import BaseModel, Field
from ragas.prompt import PydanticPrompt


class SentimentInput(BaseModel):
    text: str = Field(..., description="Text to analyze")


class SentimentOutput(BaseModel):
    sentiment: str = Field(..., description="positive, negative, or neutral")
    confidence: float = Field(..., description="Confidence score between 0 and 1")


class SentimentPrompt(PydanticPrompt[SentimentInput, SentimentOutput]):
    instruction = "Analyze the sentiment of the given text."
    input_model = SentimentInput
    output_model = SentimentOutput
    examples = [
        (
            SentimentInput(text="I love this product!"),
            SentimentOutput(sentiment="positive", confidence=0.95),
        ),
    ]

Generating Output

from ragas.llms import llm_factory

llm = llm_factory("gpt-4o-mini")

prompt = SentimentPrompt()
result = await prompt.generate(
    llm=llm,
    data=SentimentInput(text="The weather is terrible today."),
)
print(result.sentiment)     # e.g., "negative"
print(result.confidence)    # e.g., 0.88

Adapting to Another Language

spanish_prompt = await prompt.adapt(
    target_language="spanish",
    llm=llm,
    adapt_instruction=True,
)

Saving and Loading

prompt.save("my_sentiment_prompt.json")
loaded_prompt = SentimentPrompt.load("my_sentiment_prompt.json")

Internal Components

RagasOutputParser

RagasOutputParser extends LangChain's PydanticOutputParser with retry logic. When parsing fails, it invokes the FixOutputFormat prompt to ask the LLM to repair its own malformed output, decrementing the retry counter on each attempt. If all retries are exhausted, it raises a RagasOutputParserException.

FixOutputFormat

A self-contained PydanticPrompt subclass that takes the original output string and prompt text as input, and returns corrected output that satisfies the original schema constraints.

TranslateStatements

A PydanticPrompt subclass used internally by the adapt() method. It translates a list of string statements to a target language while preserving the exact count and structure of the statements. It includes explicit safeguards against instruction injection in translated text.

is_langchain_llm()

A helper function that detects whether a given LLM instance is a LangChain BaseLanguageModel (deprecated path) or a Ragas-native LLM. It emits a deprecation warning when a LangChain LLM is detected.

Related Pages

BasePrompt - Parent class providing base prompt functionality
RagasOutputParser - Output parser with retry logic for structured outputs
TranslateStatements - Prompt for multilingual adaptation of examples
BaseRagasLLM - Ragas native LLM interface used for generation
InstructorBaseRagasLLM - Instructor-based LLM interface for structured output
Vibrantlabsai_Ragas_SimplePrompt - Simpler prompt class without Pydantic validation

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment