Heuristic:NVIDIA DALI Memory Pool Tuning
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Infrastructure, GPU_Computing |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
GPU and host memory pool configuration via environment variables to control allocation strategy, growth behavior, and operator buffer presizing for stable training throughput.
Description
DALI uses memory pools for both GPU (device) and host (pinned) memory to avoid the overhead of repeated `cudaMalloc`/`cudaFree` calls. The memory pool behavior is controlled via environment variables that set the allocation backend (CUDA VMM, cudaMallocAsync, or raw cudaMalloc), buffer growth factor, and shrink thresholds. Additionally, DALI provides per-operator `bytes_per_sample_hint` and pipeline-level `enable_memory_stats` for profiling and presizing operator buffers.
Usage
Use this heuristic when observing throughput instability during training (indicating runtime memory reallocations) or when fine-tuning GPU memory usage for maximum batch size. The memory pool environment variables are global and affect all DALI pipelines in the process.
The Insight (Rule of Thumb)
- Action: Set `DALI_BUFFER_GROWTH_FACTOR=1.1` for production training to preallocate 10% extra and avoid mid-training reallocations. Use `enable_memory_stats=True` to profile operator memory needs, then set `bytes_per_sample_hint` per operator.
- Value:
- `DALI_BUFFER_GROWTH_FACTOR=1.1` (grow buffers 10% beyond current need)
- `DALI_USE_DEVICE_MEM_POOL=1` (default; use CUDA VMM pool)
- `DALI_USE_VMM=1` (prefer Virtual Memory Management over cudaMallocAsync)
- `DALI_MALLOC_POOL_THRESHOLD=32M` (default; allocations above this use pool)
- `bytes_per_sample_hint = max_reserved_memory_size * 1.1` (from `executor_statistics()`)
- Trade-off: Higher growth factor = more memory wasted but fewer reallocations. Disabling memory pools (`DALI_USE_DEVICE_MEM_POOL=0`) causes dramatic performance drop and should only be used for debugging.
Reasoning
Memory allocation is one of the most common sources of throughput instability in GPU pipelines. DALI's default memory pool (CUDA VMM) provides fast allocation by managing a virtual address range and committing physical pages as needed. The growth factor controls how much extra memory is allocated beyond the current request, reducing the frequency of page commits.
The performance ranking of allocators is: VMM > cudaMallocAsync > cudaMalloc. CUDA VMM allows over-commitment (virtual address space is reserved but physical memory is committed on demand), which is ideal for pipelines with variable-size data. cudaMallocAsync integrates with CUDA's built-in memory pool but requires CUDA 11.2+.
Operator buffer presizing via `bytes_per_sample_hint` is the most targeted optimization: it tells each operator exactly how much memory to preallocate per sample, eliminating all runtime reallocations for that operator.
Code Evidence
Memory pool environment variables from `docs/advanced_topics_performance_tuning.rst:39-134`:
DALI_HOST_BUFFER_GROWTH_FACTOR=1.0 # Default: no growth
DALI_DEVICE_BUFFER_GROWTH_FACTOR=1.0 # Grow by X factor
DALI_BUFFER_GROWTH_FACTOR=1.0 # Set both at once
DALI_HOST_BUFFER_SHRINK_THRESHOLD=0.9 # Shrink if <90% used
DALI_USE_PINNED_MEM_POOL=1 # Use pool for pinned memory
DALI_USE_DEVICE_MEM_POOL=1 # Use pool for device memory
DALI_USE_CUDA_MALLOC_ASYNC=0 # Use cudaMallocAsync instead
DALI_USE_VMM=1 # Use CUDA VMM
DALI_MALLOC_POOL_THRESHOLD=32M # Pool threshold
Warning about disabling pools from `docs/advanced_topics_performance_tuning.rst:117-123`:
WARNING: Disabling memory pools will result in dramatic drop in performance.
This should be used only for debugging purposes.
WARNING: Disabling CUDA VMM can degrade performance due to pessimistic synchronization.
Operator buffer presizing from `docs/advanced_topics_performance_tuning.rst:148-177`:
# Step 1: Enable statistics collection
pipe = create_pipeline(enable_memory_stats=True)
pipe.build()
# Step 2: Run a few iterations to collect stats
for _ in range(5):
pipe.run()
# Step 3: Query stats
stats = pipe.executor_statistics()
# Returns max_reserved_memory_size per operator
# Step 4: Use stats for presizing (add 10% headroom)
images = fn.decoders.image(
data,
bytes_per_sample_hint=int(max_reserved * 1.1)
)