Principle:Ggml org Ggml Vulkan GPU Computation

Field	Value
sources	GGML Vulkan Specification SPIR-V Specification
domains	GPU, Vulkan
last_updated	2026-02-10

Overview

Vulkan GPU Computation is the principle of accelerating tensor operations using the Vulkan compute API with SPIR-V shader compilation, providing cross-platform GPU-accelerated inference on NVIDIA, AMD, Intel, and mobile GPUs.

Description

Vulkan is a modern, low-overhead, cross-platform graphics and compute API maintained by the Khronos Group. Unlike its predecessor OpenGL, Vulkan provides explicit control over GPU resources, synchronization, and command submission, enabling higher performance through reduced driver overhead and better multi-threading support.

GGML's Vulkan backend is one of the most fully-featured GPU backends, implementing a comprehensive set of tensor operations as Vulkan compute shaders compiled to SPIR-V intermediate representation.

SPIR-V Pipeline Architecture

Shaders are authored as GLSL compute shaders in the vulkan-shaders/ directory and compiled to SPIR-V bytecode at build time. At runtime, the backend loads pre-compiled SPIR-V modules and creates VkComputePipeline objects for each operation variant. This avoids runtime shader compilation overhead.

The pipeline creation process:

• Shader module -- Created from SPIR-V bytecode (VkShaderModule)
• Pipeline layout -- Defines the descriptor set layout (buffer bindings) and push constants
• Compute pipeline -- Combines the shader module and pipeline layout into an executable pipeline

Descriptor Sets and Buffer Binding

Vulkan uses descriptor sets to bind GPU-accessible buffers to shader inputs. The GGML backend uses:

• Storage buffers -- For input tensors (weights, activations) and output tensors
• Push constants -- For small, frequently-changing parameters (dimensions, strides, scaling factors)

Command Buffer Submission

Operations are recorded into Vulkan command buffers:

• A command buffer is allocated from a command pool
• Compute dispatch commands are recorded sequentially
• Memory barriers are inserted between operations that have data dependencies
• The command buffer is submitted to a compute queue for execution

Multi-Device and Queue Management

The backend supports multiple Vulkan-capable devices (discrete GPUs, integrated GPUs). Each device context maintains its own command pool, descriptor pool, and compute queue. The backend uses the Vulkan C++ bindings (vulkan.hpp) with a dynamic dispatch loader for portability.

Shader Specialization

Many kernels use GLSL specialization constants to compile different variants at pipeline creation time, avoiding runtime branching in hot shader code. This enables efficient support for multiple quantization types and tensor layouts from the same shader source.

Usage

Apply Vulkan GPU computation when:

• Cross-platform GPU support is required (Windows, Linux, Android)
• The target GPU supports Vulkan compute (virtually all modern GPUs: NVIDIA, AMD, Intel, Qualcomm, ARM Mali)
• Low-overhead, explicit GPU control is desired
• The workload benefits from GPU parallelism (matrix multiply, element-wise operations, attention)

Vulkan is particularly well-suited for:

• Desktop inference on Windows and Linux with NVIDIA or AMD GPUs
• Android deployment where Vulkan is the standard GPU API
• Multi-GPU inference with explicit device management
• Applications that already use Vulkan for graphics and want unified GPU access

Theoretical Basis

The Vulkan compute execution model for GGML proceeds as follows:

 Initialization:
 1. Instance creation:
    Create VkInstance with required extensions (e.g., device properties)

 2. Physical device enumeration:
    Enumerate VkPhysicalDevice objects, select based on:
    - Device type (discrete GPU preferred)
    - Compute queue family availability
    - Memory heap sizes
    - Subgroup size and supported operations

 3. Logical device creation:
    Create VkDevice with compute queue and required features
    (e.g., 16-bit storage, 8-bit storage, cooperative matrix, bfloat16)

 4. Pipeline setup:
    For each operation variant:
      shader_module = vkCreateShaderModule(device, spirv_bytecode)
      pipeline_layout = vkCreatePipelineLayout(device, descriptor_layout, push_constant_range)
      pipeline = vkCreateComputePipelines(device, shader_module, pipeline_layout, specialization_info)

 5. Resource allocation:
    Allocate VkDeviceMemory and create VkBuffer for tensor storage
    Create VkDescriptorPool and allocate VkDescriptorSets

 Graph Execution:
 1. Begin command buffer:
    vkBeginCommandBuffer(cmd_buf)

 2. For each node in the computation graph:
    a. Bind pipeline:
       vkCmdBindPipeline(cmd_buf, VK_PIPELINE_BIND_POINT_COMPUTE, pipeline)

    b. Update and bind descriptors:
       Write descriptor set with buffer handles for src0, src1, dst
       vkCmdBindDescriptorSets(cmd_buf, ..., descriptor_set)

    c. Push constants:
       vkCmdPushConstants(cmd_buf, layout, stage_flags, offset, size, &params)
       -- params contains: M, N, K, strides, scale factors, etc.

    d. Dispatch compute:
       group_count_x = ceil(N / local_size_x)
       group_count_y = ceil(M / local_size_y)
       group_count_z = batch_size
       vkCmdDispatch(cmd_buf, group_count_x, group_count_y, group_count_z)

    e. Memory barrier (if next operation reads this output):
       vkCmdPipelineBarrier(cmd_buf,
         src_stage = COMPUTE, dst_stage = COMPUTE,
         buffer_barrier: src_access = SHADER_WRITE, dst_access = SHADER_READ)

 3. End and submit:
    vkEndCommandBuffer(cmd_buf)
    vkQueueSubmit(queue, cmd_buf, fence)

 4. Synchronize:
    vkWaitForFences(device, fence)

 Matrix Multiply Shader (conceptual GLSL):
 layout(local_size_x = 16, local_size_y = 16) in;

 shared float tile_A[TILE_K][TILE_M];
 shared float tile_B[TILE_K][TILE_N];

 void main() {
   // Each workgroup computes a TILE_M x TILE_N block of the output
   // Iterate over K dimension in TILE_K-sized blocks
   float acc[TILE_M/16][TILE_N/16] = 0;

   for (int k = 0; k < K; k += TILE_K) {
     // Cooperative load into shared memory
     tile_A[local_id] = A[global_row, k + ...];
     tile_B[local_id] = B[k + ..., global_col];
     barrier();

     // Multiply-accumulate using shared memory tiles
     acc += tile_A * tile_B;
     barrier();
   }

   // Write output
   C[global_row, global_col] = acc;
 }

Related Pages

• Implementation:Ggml_org_Ggml_Vulkan_backend
• Ggml_org_Ggml_Vulkan_backend -- The backend implementation that applies this principle
• Ggml_org_Ggml_Metal_GPU_Computation -- Apple-specific GPU compute alternative
• Ggml_org_Ggml_OpenCL_GPU_Computation -- Alternative cross-platform GPU compute using OpenCL
• Ggml_org_Ggml_SYCL_GPU_Computation -- Intel-focused GPU compute using SYCL