Implementation:NVIDIA DALI DynamicScratchpad

Knowledge Sources	NVIDIA_DALI
Domains	Kernels, GPU_Computing
Last Updated	2026-02-08 16:00 GMT

Overview

Implements a concrete Scratchpad that dynamically allocates temporary buffers for each memory kind using monotonic memory resources with stream-ordered allocation and deallocation.

Description

The DynamicScratchpad class is the primary concrete implementation of the abstract Scratchpad interface defined in context.h. It provides temporary memory allocation for four memory kinds: host, pinned, device, and managed. Internally it wraps monotonic memory resources (mm::monotonic_memory_resource) that grow as needed and release all memory at once when the scratchpad is destroyed.

The implementation is split into two layers. The detail::DynamicScratchpadImplT template manages the per-kind monotonic resources and optional fixed_order_resource adapters that enable stream-ordered allocation for async-capable memory kinds. The public DynamicScratchpad class inherits from both Scratchpad (for the interface) and the detail implementation (for the resource management), providing the Alloc virtual method dispatch and lazy resource initialization.

A critical design constraint is that DynamicScratchpad instances must be short-lived (typically stack-allocated local variables). Because the monotonic allocator never frees individual allocations, keeping a DynamicScratchpad alive as a class member would create an undetectable functional memory leak -- memory sanitizers would not flag it since the buffers remain reachable. The default initial allocation size per kind is 64 KiB, and resources are initialized lazily on first allocation request for each memory kind.

Usage

Use DynamicScratchpad as a local variable to provide temporary working memory to kernels. Create it with an AccessOrder (typically wrapping a CUDA stream) that governs the ordering of device memory operations. Assign it to a KernelContext::scratchpad pointer before calling kernel Run methods. The KernelManager::Run method automatically creates a DynamicScratchpad if the context does not already have one.

Code Reference

Source Location

• Repository: NVIDIA_DALI
• File: dali/kernels/dynamic_scratchpad.h
• Lines: 1-228

Signature

namespace detail {

template <typename... Kinds>
class DynamicScratchpadImplT {
 protected:
  template <typename Kind>
  void set_upstream_resource(mm::memory_resource<Kind> *rsrc);

  template <typename Kind>
  void set_upstream_resource(mm::async_memory_resource<Kind> *rsrc,
                             AccessOrder alloc_order,
                             AccessOrder dealloc_order = {});

  template <typename Kind>
  size_t &initial_size();

  template <typename Kind>
  mm::memory_resource<Kind> *get_upstream() const;
};

using DynamicScratchpadImpl = DynamicScratchpadImplT<
      mm::memory_kind::host,
      mm::memory_kind::pinned,
      mm::memory_kind::device,
      mm::memory_kind::managed>;

}  // namespace detail

class DynamicScratchpad
  : public Scratchpad
  , private detail::DynamicScratchpadImpl {
 public:
  using scratch_sizes_t = std::array<size_t, static_cast<size_t>(mm::memory_kind_id::count)>;

  explicit DynamicScratchpad(AccessOrder device_order,
                             AccessOrder pinned_dealloc_order = {},
                             AccessOrder managed_dealloc_order = {},
                             scratch_sizes_t initial_sizes = {});

  virtual void *Alloc(mm::memory_kind_id kind_id, size_t bytes, size_t alignment);

  template <typename Kind>
  void *AllocImpl(size_t bytes, size_t alignment);
};

Import

#include "dali/kernels/dynamic_scratchpad.h"

I/O Contract

Inputs

Name	Type	Required	Description
device_order	AccessOrder	Yes	Allocation and deallocation order for device memory (typically wraps a CUDA stream)
pinned_dealloc_order	AccessOrder	No	Deallocation order for pinned memory; defaults to device_order
managed_dealloc_order	AccessOrder	No	Deallocation order for managed memory; defaults to device_order
initial_sizes	scratch_sizes_t	No	Initial buffer sizes per memory kind in bytes; defaults to 64 KiB each
kind_id	mm::memory_kind_id	Yes	Memory kind identifier for Alloc dispatch (host, pinned, device, managed)
bytes	size_t	Yes	Number of bytes to allocate
alignment	size_t	Yes	Required alignment of the allocation

Outputs

Name	Type	Description
(pointer)	void*	Pointer to allocated memory of the requested kind, or nullptr for zero-sized allocations

Usage Examples

Basic Dynamic Scratchpad Usage

#include "dali/kernels/dynamic_scratchpad.h"
#include "dali/kernels/context.h"

void process(cudaStream_t stream) {
  // Create a scratchpad scoped to this function
  DynamicScratchpad scratch(AccessOrder(stream));

  // Allocate pinned host memory for staging
  char *pinned = scratch.Allocate<mm::memory_kind::pinned, char>(4096);

  // Allocate device memory
  float *gpu_buf = scratch.AllocateGPU<float>(1024);

  // Use in a kernel context
  KernelContext ctx;
  ctx.gpu.stream = stream;
  ctx.scratchpad = &scratch;

  // All memory freed when scratch goes out of scope
}

Providing Custom Initial Sizes

#include "dali/kernels/dynamic_scratchpad.h"

void process_large(cudaStream_t stream) {
  DynamicScratchpad::scratch_sizes_t sizes = {
    1 << 20,  // 1 MiB host
    1 << 20,  // 1 MiB pinned
    4 << 20,  // 4 MiB device
    0         // managed (will default to 64 KiB)
  };
  DynamicScratchpad scratch(AccessOrder(stream), {}, {}, sizes);

  float *gpu_data = scratch.AllocateGPU<float>(256 * 1024);
}

Related Pages

• Environment:NVIDIA_DALI_CUDA_GPU_Environment