Principle:Eventual Inc Daft UDF Metrics
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Observability |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Technique for instrumenting user-defined functions with custom counter metrics.
Description
UDF metrics allow tracking custom counters within UDF execution for monitoring, debugging, and performance analysis. Metrics are collected per-operator and aggregated across partitions. The system uses a context variable to track the current operator's metrics context, ensuring that metrics are only recorded during active UDF execution. Counters support optional descriptions and key-value attribute tags for categorization.
Usage
Use UDF metrics when you need to track custom metrics within UDF code, such as cache hit/miss rates, external API call counts, error rates, or any custom counters relevant to your processing logic.
Theoretical Basis
Operator-scoped metric collection with counter semantics and attribute-based tagging. The metrics context is thread-safe via context variables:
# UDF execution framework sets up metrics context
with metrics_context() as operator_metrics:
# User code calls increment_counter
user_function(inputs)
# Metrics are collected in operator_metrics
# After execution, operator_metrics contains aggregated counters
# accessible by name and attributes
Calling increment_counter outside a UDF execution context issues a runtime warning and discards the metric.