View source for Principle:Ggml org Ggml CPU Compute Engine

{{PageInfo|type=Principle|title=Ggml_org_Ggml_CPU_Compute_Engine}}
{{PageInfo|type=Principle|title=Ggml_org_Ggml_CPU_Compute_Engine}}
{| class="wikitable"
|-
! Field !! Value
|-
| '''sources''' || [https://github.com/ggml-org/ggml GGML]
|-
| '''domains''' || CPU, Parallelism
|-
| '''last_updated''' || 2026-02-10
|}

== Overview ==

The CPU Compute Engine is the principle of executing tensor computation graphs on general-purpose CPUs using a multi-threaded work-distribution model with per-operation parallelism across rows and batch dimensions.

=== Description ===

The CPU backend is GGML's foundational compute engine. It provides implementations for every tensor operation in the GGML operation set, serving both as the primary execution path on CPU-only systems and as the fallback for operations not supported by accelerator backends.

The CPU compute engine is built around several key design principles:

'''Multi-threaded Graph Execution'''

When a computation graph is submitted, the engine distributes work across a configurable thread pool. The threading model is operation-centric: for each node in the graph, all available threads cooperate to compute that single operation before moving to the next node. This is coordinated through a barrier-based synchronization mechanism:

* Thread 0 (the main thread) iterates over graph nodes and initiates each operation
* Worker threads wait at a barrier, are woken for each operation, compute their assigned portion, and re-enter the barrier
* Work is partitioned by rows or by batch slices depending on the operation

'''Operation Dispatch'''

Each GGML operation type has a dedicated compute function. The dispatch table maps operation enums to their implementations. Operations include:
* Matrix multiply (MUL_MAT) with specialized paths for different quantization types
* Element-wise operations (ADD, MUL, GELU, SILU, etc.)
* Tensor manipulation (RESHAPE, PERMUTE, TRANSPOSE, etc.)
* Attention-specific operations (FLASH_ATTN, ROPE, etc.)

'''Extra Buffer Types and Repacking'''

The CPU backend supports "extra" buffer types that enable optimized data layouts. For example, the repack buffer type rearranges weight matrices into formats that are more cache-friendly for the CPU's SIMD multiply kernels (e.g., blocked layouts for AMX instructions, or KleidiAI-optimized formats for ARM).

'''Integration with llamafile SGEMM'''

On supported platforms, the CPU backend can delegate matrix multiplication to llamafile's SGEMM implementation, which provides its own architecture-tuned matrix multiply kernels as an alternative to hand-written GGML kernels.

=== Usage ===

The CPU compute engine applies in all GGML deployments because:

* It is always available as a fallback -- every operation has a CPU implementation
* It is the primary backend when no GPU or accelerator is present
* Even when accelerators are used, some operations may be offloaded back to CPU if the accelerator does not support them
* It supports the full range of quantization types, including mixed-precision operations

Configuration knobs include:

* '''Thread count''' -- Number of worker threads (n_threads)
* '''Thread affinity''' -- CPU core pinning (threadpool configuration)
* '''Extra buffer types''' -- Enabling AMX, KleidiAI, or repack optimizations
* '''OpenMP vs. built-in threadpool''' -- The engine supports both OpenMP-based and custom threadpool-based parallelism

== Theoretical Basis ==

The CPU compute engine's execution model can be described as follows:

  '''Input:''' Computation graph G with N nodes, thread pool with T threads

  '''Graph Execution Loop (Thread 0):'''
  For each node i in G (topological order):
    1. '''Select compute function''' based on node.op
    2. '''Determine parallelism:'''
       n_tasks = compute_forward_get_n_tasks(node)
       -- For MUL_MAT: n_tasks = T (parallelize across output rows)
       -- For element-wise ops: n_tasks = T (parallelize across elements)
       -- For some ops: n_tasks = 1 (serial execution)
    3. '''Signal worker threads''' via barrier
    4. '''All T threads execute:'''
       For thread_id in [0..n_tasks-1]:
         compute_forward(node, thread_id, n_tasks)
         -- Each thread computes rows [thread_id * rows/n_tasks .. (thread_id+1) * rows/n_tasks)
    5. '''Barrier''' -- Wait for all threads to complete this node
    6. '''Proceed to next node'''

  '''Work Partitioning for MUL_MAT (example):'''
  Given: C[M x N] = A[M x K] * B[K x N]
  Thread t computes rows:
    row_start = (t * M) / n_tasks
    row_end   = ((t+1) * M) / n_tasks
  Each thread computes C[row_start..row_end, :] independently

  '''Barrier Synchronization:'''
  Uses a spin-wait barrier with cache-line-aligned counters:
    - Workers spin on a shared counter
    - Thread 0 increments the counter to release workers
    - Workers decrement a completion counter when done
    - Thread 0 spins on the completion counter

This model avoids the overhead of per-operation thread creation and minimizes synchronization costs by using a persistent thread pool with simple barrier coordination.

== Related Pages ==

* [[implemented_by::Implementation:Ggml_org_Ggml_Cpu_compute_engine]]
* [[Ggml_org_Ggml_Cpu_compute_engine]] -- The implementation of this principle
* [[Ggml_org_Ggml_Architecture_Specific_SIMD_Quantization]] -- SIMD-optimized kernels used by the CPU compute engine
* [[Ggml_org_Ggml_BLAS_Matrix_Multiplication]] -- Alternative matrix multiply path using BLAS libraries

[[Category:Principles]]

[[Category:Principles]]