Implementation:Ggml org Llama cpp Q8dot Benchmark

Knowledge Sources	Ggml_org_Llama_cpp
Domains	Quantization, Benchmarking
Last Updated	2026-02-15 00:00 GMT

Overview

Benchmarks quantized Q4-Q8 dot product performance, comparing a simple scalar implementation against ggml's optimized vec_dot routines.

Description

This proof-of-concept program generates random Q4_0 or Q4_1 quantized blocks and Q8_0 blocks, then computes dot products using both a naive scalar loop and ggml's `vec_dot` function. It measures execution time over multiple iterations, collecting statistics (mean, standard deviation, max) via the `Stat` helper class and reporting average dot product values to prevent compiler dead-code elimination.

Usage

Use this program to validate and compare the performance of ggml's quantized dot product kernels against straightforward reference implementations. It accepts command-line arguments for loop count and quantization type.

Code Reference

Source Location

• Repository: Ggml_org_Llama_cpp
• File: pocs/vdot/q8dot.cpp
• Lines: 1-173

Signature

int main(int argc, char** argv);

// Helper functions
template <typename T>
static void fillQ4blocks(std::vector<T>& blocks, std::mt19937& rndm);
static void fillQ80blocks(std::vector<block_q8_0>& blocks, std::mt19937& rndm);
static float simpleDot(const block_q4_0& x, const block_q8_0& y);
static float simpleDot(const block_q4_1& x, const block_q8_0& y);

struct Stat {
    void addResult(double s, double t);
    void reportResult(const char* title) const;
};

Import

#include <cstdio>
#include <type_traits>
#include <vector>
#include <random>
#include <chrono>
#include <cstdlib>
#include <cmath>
#include <cassert>
#include <cstring>
#include <array>

#include <ggml.h>
#include <ggml-cpu.h>

I/O Contract

Inputs

Name	Type	Required	Description
argc	int	Yes	Command-line argument count
argv[1]	char *	No	Number of benchmark iterations (default: 10)
argv[2]	char *	No	Quantization type: 0 for Q4_0, 1 for Q4_1 (default: 1)

Outputs

Name	Type	Description
return value	int	0 on success
stdout	text	Benchmark results with mean dot product, mean/stddev/max time for both simple and ggml implementations

Usage Examples

# Run 100 iterations benchmarking Q4_1 dot product
./q8dot 100 1

# Run 50 iterations benchmarking Q4_0 dot product
./q8dot 50 0

Related Pages

• Principle:Ggml_org_Llama_cpp_Quantization