Principle:Eventual Inc Daft Class UDF
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, User_Defined_Functions |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Technique for creating stateful user-defined functions with expensive one-time initialization.
Description
Class UDFs wrap a Python class with __init__ for initialization (e.g., loading ML models, establishing database connections) and methods for processing. Initialization is amortized across rows because each class instance is created once and reused for an entire partition or set of partitions. Class UDFs support GPU resource requests via the gpus parameter, concurrency control via max_concurrency, and multiple methods per class. Methods can be row-wise, async, generator, or batch variants using the @daft.method decorator.
Usage
Use class UDFs when you need stateful processing with expensive initialization, such as ML model loading, database connection pooling, or any scenario where setup cost should be amortized across many rows.
Theoretical Basis
Class UDFs follow an actor-based processing pattern where initialization cost is amortized across many invocations. The lifecycle is:
1. Lazy initialization: arguments saved at decoration time
2. Instance creation: __init__ called once per worker at execution time
3. Processing: methods called repeatedly on the same instance
4. Cleanup: instance garbage collected when partition processing completes
This pattern is particularly valuable when the initialization cost (e.g., loading a 10GB model) vastly exceeds per-row processing cost.