Principle:Ggml org Ggml CI Build Testing
| Knowledge Sources | |
|---|---|
| Domains | CI_CD, Testing |
| Last Updated | 2026-02-10 |
Overview
CI Build Testing is the continuous integration pipeline that validates multi-backend builds, runs test suites, and executes example model inference across hardware configurations.
Description
GGML supports numerous hardware backends -- CPU, CUDA, ROCm/HIP, SYCL, Vulkan, Metal, WebGPU, and MUSA -- and each combination of backend, build type, and platform can introduce regressions. CI Build Testing codifies a reproducible, automated pipeline (implemented as a Bash script at ci/run.sh) that:
- Configures the build by setting CMake flags based on environment variables (
GG_BUILD_CUDA,GG_BUILD_SYCL,GG_BUILD_VULKAN,GG_BUILD_ROCM,GG_BUILD_METAL,GG_BUILD_WEBGPU,GG_BUILD_MUSA). Each flag activates the corresponding GGML backend and applies any required vendor-specific settings (e.g., queryingnvidia-smifor CUDA architecture, requiringONEAPI_ROOTfor SYCL). - Builds in both Debug and Release modes using CMake and Make, exercising compiler warnings and assertions (Debug) as well as optimization correctness (Release).
- Runs the CTest suite to validate unit tests and backend operation tests. In Debug mode, expensive tests like
test-optandtest-backend-opsare excluded; in Release mode, the full suite runs unless a low-performance flag is set. - Executes example model inference for GPT-2, SAM (Segment Anything Model), and YOLO (object detection). These integration tests download pre-trained weights, run inference with fixed seeds, and validate output against expected patterns using
grepassertions. This catches regressions in the full stack from model loading through graph construction to backend computation. - Produces structured logs and exit codes for each stage, written to an output directory with a summary README in Markdown format.
The pipeline also establishes a Python virtual environment and installs project dependencies from requirements.txt, ensuring that Python-dependent conversion scripts and model preparation steps work correctly.
Usage
Apply this principle when contributing changes to GGML that affect the build system, backend implementations, or core tensor operations. Run bash ./ci/run.sh ./tmp/results ./tmp/mnt locally before submitting changes to catch build failures and test regressions. Set the appropriate GG_BUILD_* environment variables to test specific backend configurations. Use GG_BUILD_LOW_PERF=1 on resource-constrained machines to skip long-running optimization tests.
Theoretical Basis
The CI pipeline embodies several core principles of continuous integration for multi-platform native software:
- Backend-Conditional Compilation -- Each hardware backend is gated behind a compile-time flag that maps to a CMake option. The CI script mirrors this structure by translating environment variables to CMake arguments. This ensures that every supported backend combination is testable in isolation and that enabling one backend does not break another.
- Two-Tier Testing Strategy -- The pipeline distinguishes between unit tests (CTest) and integration tests (model inference). Unit tests validate individual operations and APIs in isolation; integration tests validate the end-to-end path from model files to inference output. Both tiers are necessary because unit-test passes do not guarantee correct behavior when components are composed, and integration tests alone cannot pinpoint the source of failures.
- Deterministic Reproducibility -- Model inference tests use fixed random seeds (
-s 1234) and known input prompts, so their outputs are deterministic across runs. This allows output validation with simple pattern matching rather than statistical thresholds. - Output Validation by Assertion -- Integration tests validate results using
grepagainst expected output patterns (e.g., checking that YOLO detects a "dog" with 55-59% confidence). This is a form of golden-output testing where known-good outputs serve as regression baselines. - Graceful Degradation -- The pipeline supports a
GG_BUILD_LOW_PERFflag to skip expensive tests on slower machines, and individual test stages are sequenced with early exit on failure (test $ret -eq 0 && gg_run ...). This prevents wasting compute on downstream stages when an earlier stage has already failed.