Principle:Ggml org Ggml SIMD Abstraction

Attribute	Value
Page Type	Principle
Full Name	Ggml_org_Ggml_SIMD_Abstraction
Short Name	SIMD_Abstraction
Domain Tags	SIMD, Portability
Knowledge Source	GGML
Last Updated	2026-02-10

Overview

Providing a platform-agnostic macro layer over hardware SIMD intrinsics (ARM NEON, x86 AVX/SSE, POWER VSX, s390x VXE, RISC-V RVV, WebAssembly SIMD) so that higher-level tensor operations can be written once and compiled efficiently for any supported architecture.

Description

SIMD Abstraction is the principle of defining a unified set of macros and inline functions that map common SIMD (Single Instruction, Multiple Data) operations to the native intrinsics of each supported CPU architecture. Rather than writing separate implementations of every tensor operation for each architecture, GGML defines a portable abstraction layer in simd-mappings.h that translates operations like "convert FP16 to FP32", "load a vector of floats", or "compute a dot product" into the appropriate hardware-specific instructions.

The abstraction layer covers the following architectures:

Architecture	Intrinsics	Key Features
ARM	NEON, SVE	__fp16 native type, vld1q_f32, vmlaq_f32
x86	SSE, AVX, AVX2, AVX-512, F16C, FMA	_mm256_loadu_ps, _mm256_fmadd_ps
POWER	VSX, POWER9	Inline assembly for FP16 conversion
s390x	VXE, VXE2	IBM Z vector extensions
RISC-V	RVV (Vector Extension)	Scalable vector length via riscv_vector.h
WebAssembly	WASM SIMD128	128-bit SIMD in browser environments

The primary abstractions include:

• FP16/FP32 conversion: GGML_CPU_COMPUTE_FP16_TO_FP32(x) and GGML_CPU_COMPUTE_FP32_TO_FP16(x) map to __fp16 casts on ARM, _cvtsh_ss/_cvtss_sh on x86 with F16C, inline assembly on POWER9, and lookup-table-based conversion as a fallback.
• FP16 lookup table fallback: GGML_CPU_FP16_TO_FP32(x) uses a precomputed 64K-entry lookup table on architectures without native FP16 conversion, providing O(1) conversion at the cost of 256 KB of table memory.
• BF16 conversion: GGML_CPU_BF16_TO_FP32(x) converts bfloat16 to float32 via bit manipulation (left-shift by 16), with ARM-specific vcvtah_f32_bf16 support.

Usage

SIMD abstraction is used throughout the CPU backend:

• Vectorized operations: The vector operation functions (ggml_vec_dot_f32, ggml_vec_add_f32, etc.) use these abstractions to write SIMD-accelerated loops that compile correctly on all platforms.
• Quantization kernels: Dequantization and quantized dot product functions use the SIMD abstractions for the inner loops that process quantized blocks.
• Type conversion: Every place in the codebase that needs to convert between FP16/BF16 and FP32 uses the abstraction macros rather than direct intrinsics, ensuring portability.
• New architecture support: Adding support for a new CPU architecture requires only extending the conditional compilation blocks in simd-mappings.h, without modifying the hundreds of functions that use the abstractions.

Theoretical Basis

SIMD Programming Model

SIMD (Single Instruction, Multiple Data) is a parallel processing paradigm where a single instruction operates on multiple data elements simultaneously. Modern CPUs provide SIMD through vector registers and vector instructions: ARM NEON provides 128-bit registers (4 floats), AVX2 provides 256-bit registers (8 floats), AVX-512 provides 512-bit registers (16 floats), and RISC-V RVV provides scalable-length registers. Exploiting SIMD is critical for numerical computation performance, as it can provide 4-16x throughput improvement for data-parallel operations.

Abstraction by Conditional Compilation

GGML's approach uses C preprocessor conditional compilation (#if defined(__ARM_NEON), #elif defined(__F16C__), etc.) to select the appropriate implementation at compile time. This is a zero-cost abstraction: the preprocessor resolves all conditionals before compilation, so the generated code contains only the native intrinsics for the target architecture. There is no runtime dispatch overhead and no virtual function calls -- just direct hardware instructions.

FP16 Conversion as a Critical Path

In quantized inference, FP16 conversion is on the critical path because quantization scales are stored as FP16 values that must be converted to FP32 for computation. The abstraction layer provides three tiers of FP16 conversion performance:

1. Hardware instruction: Direct FP16 support (ARM __fp16, x86 F16C) provides single-instruction conversion.
2. Inline assembly: On POWER9, inline assembly accesses the FP16 conversion unit without C language support for the __fp16 type.
3. Lookup table: On architectures without FP16 hardware, a 64K-entry table provides constant-time conversion with a single memory lookup per value.

Platform Detection

The abstraction layer relies on compiler-defined macros (__ARM_NEON, __AVX2__, __POWER9_VECTOR__, __riscv_v_intrinsic, __VXE__) that are automatically set when the compiler targets a specific architecture with the appropriate flags. This ensures that the correct code path is selected based on the actual compilation target, not runtime detection, eliminating branch misprediction costs in hot loops.

Related Pages

Implemented By

• Implementation:Ggml_org_Ggml_Cpu_simd_mappings