Implementation:Confident ai Deepeval ConfidentInstrumentationSettings

Metadata

Knowledge Sources	DeepEval
Domains	LLM_Evaluation AI_Agents
Last Updated	2026-02-14 09:00 GMT

Overview

Concrete instrumentation settings class that integrates DeepEval with PydanticAI agents. The ConfidentInstrumentationSettings class extends PydanticAI's InstrumentationSettings and installs custom OTEL span processors to capture agent execution traces for evaluation.

Description

The ConfidentInstrumentationSettings is passed directly to a PydanticAI Agent constructor via the instrumentation_settings parameter. Once configured, it intercepts all OpenTelemetry spans emitted during agent execution (model calls, tool invocations, agent steps) and translates them into DeepEval's trace format.

Key capabilities:

• OTEL-based trace capture -- intercepts spans emitted by PydanticAI's built-in OTEL instrumentation.
• Agent-level metric evaluation -- when agent_metrics are provided, evaluation runs automatically after each invocation.
• Metric collection support -- can reference a named metric collection configured on the Confident AI platform.
• Metadata and tagging -- supports arbitrary metadata and tags for trace organization.

Usage

Import and pass to the PydanticAI Agent constructor:

from deepeval.integrations.pydantic_ai import ConfidentInstrumentationSettings

Code Reference

Source Location

• Repository: confident-ai/deepeval
• File: deepeval/integrations/pydantic_ai/instrumentator.py (lines 117--196)

Signature

class ConfidentInstrumentationSettings(InstrumentationSettings):
    def __init__(
        self,
        api_key: Optional[str] = None,
        name: Optional[str] = None,
        thread_id: Optional[str] = None,
        user_id: Optional[str] = None,
        metadata: Optional[Dict[str, Any]] = None,
        tags: Optional[List[str]] = None,
        metric_collection: Optional[str] = None,
        agent_metrics: Optional[List[BaseMetric]] = None,
        ...
    ):
        ...

Import

from deepeval.integrations.pydantic_ai import ConfidentInstrumentationSettings

Parent Class

• InstrumentationSettings from pydantic_ai.models.instrumented

I/O Contract

Inputs (Constructor Parameters)

Input Contract

Name	Type	Default	Description
api_key	Optional[str]	None	Confident AI API key. Falls back to environment variable if not provided.
name	Optional[str]	None	Human-readable name for the traced agent.
thread_id	Optional[str]	None	Conversation thread identifier for grouping related invocations.
user_id	Optional[str]	None	Identifier for the end user associated with this trace.
metadata	Optional[Dict[str, Any]]	None	Arbitrary key-value metadata attached to the trace.
tags	Optional[List[str]]	None	Tags for categorizing and filtering traces.
metric_collection	Optional[str]	None	Name of a pre-defined metric collection on the Confident AI platform.
agent_metrics	Optional[List[BaseMetric]]	None	List of agent-level evaluation metrics to run automatically after each invocation.

Outputs

Output Contract

Name	Type	Description
Trace	Internal trace object	Hierarchical trace of the PydanticAI agent execution, captured via OTEL span processing.
Metric results	Evaluation scores	When agent_metrics are configured, evaluation results are computed and optionally pushed to the Confident AI platform.

Usage Examples

Example 1: Basic PydanticAI Agent Instrumentation

Attach instrumentation settings to a PydanticAI agent with automatic metric evaluation.

from deepeval.integrations.pydantic_ai import ConfidentInstrumentationSettings
from deepeval.metrics import TaskCompletionMetric
from pydantic_ai import Agent

settings = ConfidentInstrumentationSettings(
    name="my-pydantic-agent",
    agent_metrics=[TaskCompletionMetric()],
)
agent = Agent(model, instrumentation_settings=settings)

• The settings object is passed via PydanticAI's instrumentation_settings parameter.
• After each agent invocation, the TaskCompletionMetric is evaluated against the captured OTEL trace.

Related Pages

• Principle:Confident_ai_Deepeval_PydanticAI_Instrumentation