Implementation:Confident ai Deepeval LLMTestCase

Overview

LLMTestCase is a Pydantic BaseModel class in the deepeval library that structures LLM inputs, outputs, and contextual information into a standardized test case object. It serves as the primary data container consumed by all evaluation metrics and evaluation functions in the framework.

This is an API Doc implementation.

Source

Repository: Confident AI Deepeval
File: deepeval/test_case/llm_test_case.py, lines 301-543
Class: LLMTestCase (extends Pydantic BaseModel)

Import

from deepeval.test_case import LLMTestCase

Constructor Signature

LLMTestCase(
    input: str,
    actual_output: Optional[str] = None,
    expected_output: Optional[str] = None,
    context: Optional[List[str]] = None,
    retrieval_context: Optional[List[str]] = None,
    tools_called: Optional[List[ToolCall]] = None,
    expected_tools: Optional[List[ToolCall]] = None,
    additional_metadata: Optional[Dict] = None
)

Parameters

Parameter	Type	Required	Default	Description
`input`	`str`	Yes	--	The user's input prompt or query sent to the LLM. This is the only required field.
`actual_output`	`Optional[str]`	No	`None`	The LLM's actual response to the input. Required by most evaluation metrics.
`expected_output`	`Optional[str]`	No	`None`	The expected or reference response. Used by correctness and similarity metrics.
`context`	`Optional[List[str]]`	No	`None`	The ground-truth context that should ideally be used. Used for evaluating retrieval quality.
`retrieval_context`	`Optional[List[str]]`	No	`None`	The context that was actually retrieved and provided to the LLM. Required by faithfulness and contextual relevancy metrics.
`tools_called`	`Optional[List[ToolCall]]`	No	`None`	The tools that were actually called by the LLM during execution. Used for tool use evaluation.
`expected_tools`	`Optional[List[ToolCall]]`	No	`None`	The tools that should have been called. Used for tool use correctness evaluation.
`additional_metadata`	`Optional[Dict]`	No	`None`	Arbitrary key-value metadata to attach to the test case for tracking and filtering.

Input / Output

Inputs: The constructor parameters described above.
Outputs: A configured LLMTestCase object that can be passed to evaluate(), assert_test(), or individual metric measure() methods.

Example

Basic QA Test Case

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is Python?",
    actual_output="Python is a programming language.",
    expected_output="Python is a high-level programming language."
)

RAG Test Case with Retrieval Context

from deepeval.test_case import LLMTestCase

rag_test_case = LLMTestCase(
    input="What are the side effects of aspirin?",
    actual_output="Common side effects of aspirin include stomach upset and increased bleeding risk.",
    retrieval_context=[
        "Aspirin may cause stomach irritation, nausea, and gastrointestinal bleeding.",
        "Aspirin inhibits platelet aggregation, which can increase bleeding risk."
    ],
    context=[
        "Aspirin (acetylsalicylic acid) is a nonsteroidal anti-inflammatory drug.",
        "Common side effects include GI irritation, bleeding, and allergic reactions."
    ]
)

Test Case with Metadata

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="Summarize the quarterly report.",
    actual_output="Revenue increased by 15% year-over-year...",
    additional_metadata={
        "model_version": "gpt-4o-2024-05",
        "prompt_template": "v2.1",
        "category": "summarization"
    }
)

Field Requirements by Metric

Different metrics require different subsets of fields:

Metric	Required Fields
AnswerRelevancyMetric	`input`, `actual_output`
FaithfulnessMetric	`input`, `actual_output`, `retrieval_context`
GEval	Depends on `evaluation_params` configuration
ContextualRelevancyMetric	`input`, `actual_output`, `retrieval_context`

Metadata

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment