Principle:Confident ai Deepeval PydanticAI Instrumentation
| Knowledge Sources | |
|---|---|
| Domains | |
| Last Updated | 2026-02-14 09:00 GMT |
Overview
A design principle for instrumenting PydanticAI agents via OpenTelemetry-based instrumentation settings. Rather than using framework-specific callbacks, PydanticAI exposes an InstrumentationSettings interface that leverages OTEL span processors to capture agent execution traces in a standardized telemetry format.
Description
PydanticAI is an agent framework built on top of Pydantic's type system. Its instrumentation model differs from callback-based frameworks (like LangChain) by adopting the OpenTelemetry (OTEL) standard for trace collection. PydanticAI agents accept an instrumentation_settings parameter that configures how execution spans are captured and processed.
DeepEval's integration with PydanticAI works by providing a custom InstrumentationSettings subclass that:
- Installs OTEL span processors that intercept PydanticAI execution spans (model calls, tool invocations, agent lifecycle events).
- Translates OTEL spans into DeepEval's internal trace representation for evaluation.
- Supports metric evaluation by attaching agent-level metrics that run automatically after each agent invocation.
This approach is architecturally distinct from callback-based instrumentation because it relies on the OpenTelemetry protocol as the intermediary data format, making it compatible with broader observability tooling while still feeding into DeepEval's evaluation pipeline.
Usage
PydanticAI instrumentation is used when:
- A PydanticAI agent needs to be evaluated using DeepEval metrics.
- Developers want to capture traces from PydanticAI agents for offline analysis.
- The OTEL-based tracing infrastructure is preferred over custom callbacks.
The general pattern is:
PYDANTICAI_INSTRUMENTATION(agent A):
1. CREATE instrumentation settings with DeepEval's custom settings class
2. CONFIGURE agent-level metrics and metadata
3. PASS settings to PydanticAI Agent constructor via instrumentation_settings parameter
4. On each agent invocation:
a. PydanticAI emits OTEL spans for model calls and tool uses
b. DeepEval's span processor captures and translates spans
c. Evaluation metrics run against the collected trace
Theoretical Basis
This principle is grounded in:
- OpenTelemetry instrumentation -- leverages the OTEL standard for distributed tracing, providing a vendor-neutral protocol for capturing execution spans. This ensures compatibility with existing observability infrastructure.
- Span processor pattern -- OTEL span processors act as middleware in the tracing pipeline, receiving span data as it is emitted and transforming or forwarding it. DeepEval's processor transforms OTEL spans into evaluation-ready trace data.
The key advantage of this approach over callback-based instrumentation is standards compliance. By building on OTEL, the instrumentation automatically benefits from the ecosystem of OTEL exporters, processors, and tooling while serving DeepEval's specific evaluation needs.