Principle:Eventual Inc Daft Resource Configuration
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Resource_Management |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Technique for specifying computational resource requirements for UDF execution.
Description
Resource configuration declares CPU, GPU, and memory requirements for UDF tasks, enabling the scheduler to allocate appropriate resources in distributed environments. Each ResourceRequest specifies optional CPU count, GPU count, and memory byte limits. These declarations are critical for GPU-accelerated inference workloads where tasks must be co-located with available hardware.
Usage
Use resource configuration when UDFs require specific hardware resources (GPUs, memory) for execution. This is essential for ML model inference on GPUs, memory-intensive operations, and controlling parallelism in distributed clusters.
Theoretical Basis
Resource-aware scheduling where task resource declarations guide placement and concurrency decisions. The scheduler uses a bin-packing algorithm to assign tasks to workers based on available resources:
for each task T with ResourceRequest R:
find worker W where:
W.available_cpus >= R.num_cpus
W.available_gpus >= R.num_gpus
W.available_memory >= R.memory_bytes
assign T to W
W.available_resources -= R
The max_resources static method computes field-wise maximums across multiple resource requests for planning purposes.