Principle:Ggml org Ggml Graph Memory Allocation
Summary
Graph Memory Allocation is the principle of pre-computing and optimizing memory layout for a computation graph before execution. By analyzing the full graph structure ahead of time, the allocator determines which tensors are live at each step and assigns memory regions so that intermediate tensors that are no longer needed can share the same memory. This separation of memory planning from computation enables optimal memory utilization.
Core Idea
Rather than allocating memory on-the-fly as each operation executes, the entire computation graph is inspected first. A memory plan is produced that maps every tensor to a specific offset within one or more pre-allocated buffers. Tensors whose lifetimes do not overlap are assigned to overlapping memory regions, dramatically reducing peak memory consumption.
Theory
The problem is closely analogous to register allocation in compilers:
- Liveness analysis — For each tensor in the graph, determine the range of operations during which its value is needed (from its producing operation to its last consumer).
- Memory planning — Using the liveness intervals, compute a mapping from tensors to buffer offsets. Tensors with non-overlapping lifetimes may reuse the same memory, similar to how a compiler reuses registers for variables with non-overlapping live ranges.
- Buffer assignment — Assign each tensor to a concrete region within one or more backend buffers according to the plan.
Key Properties
- Memory reuse — Intermediate tensors that are no longer needed by any downstream operation can share memory with newly produced tensors, reducing peak allocation.
- Separation of planning and execution — The memory layout is fully determined before any computation runs. This allows the allocator to make globally optimal decisions rather than greedy local ones.
- Backend awareness — In multi-backend scenarios (e.g., CPU + GPU), the planner accounts for which backend each tensor resides on and allocates from the appropriate buffer type.