Implementation:Ggml org Ggml Sycl vecdotq
| Knowledge Sources | |
|---|---|
| Domains | ML_Infrastructure, GPU_Compute, Quantization |
| Last Updated | 2025-05-15 12:00 GMT |
Overview
Inline vector dot product functions between quantized blocks and q8_1 blocks for all quantization types on SYCL, serving as the core compute primitives for MMQ and MMVQ kernels.
Description
vecdotq.hpp provides the fundamental quantized dot product operations used by the matrix multiplication kernels in the SYCL backend. Each function computes the dot product between a quantized weight block and a q8_1-quantized input block without full dequantization, operating directly on the packed integer representations. The header includes:
- Standard vec_dot functions: vec_dot_q4_0_q8_1, vec_dot_q4_1_q8_1, vec_dot_q5_0_q8_1, vec_dot_q5_1_q8_1, vec_dot_q8_0_q8_1 -- each unpacking the specific quantization format, multiplying with q8_1 values, and accumulating partial sums.
- K-quant vec_dot functions: vec_dot_q2_K_q8_1, vec_dot_q3_K_q8_1, vec_dot_q4_K_q8_1, vec_dot_q5_K_q8_1, vec_dot_q6_K_q8_1 -- handling the more complex K-quant block layouts with multiple scale groups.
- Reorder-aware variants: Versions that work with Structure-of-Arrays memory layouts where scales and quantized values are stored in separate arrays for better GPU memory access patterns.
- Helper functions: get_int_b1 (byte-to-int packing), get_int_from_int8/get_int_from_uint8 (aligned and unaligned variants), get_int_from_table_16 (lookup table for IQ types) -- all declared static __dpct_inline__ for maximum performance.
The vec_dot_q_sycl_t function pointer typedef allows these functions to be passed as template parameters to the MMQ and MMVQ kernel templates.
Usage
Included by mmq.cpp and mmvq.cpp. These functions are called from within SYCL kernel parallel_for bodies, with each work-item computing one or more quantized dot products as part of the tiled matrix multiplication algorithm.
Code Reference
Source Location
- Repository: GGML
- File: src/ggml-sycl/vecdotq.hpp
- Lines: 1361
Signatures
// Function pointer type
typedef float (*vec_dot_q_sycl_t)(const void * __restrict__ vbq,
const block_q8_1 * __restrict__ bq8_1, const int & iqs);
// Helper functions
static __dpct_inline__ int get_int_b1(const void * x, const int & i32);
static __dpct_inline__ int get_int_from_int8(const int8_t* x8, const int& i32);
static __dpct_inline__ int get_int_from_uint8(const uint8_t* x8, const int& i32);
static __dpct_inline__ int get_int_from_int8_aligned(const int8_t* x8, const int& i32);
static __dpct_inline__ int get_int_from_uint8_aligned(const uint8_t* x8, const int& i32);
static __dpct_inline__ void get_int_from_table_16(const uint32_t &q4,
const uint8_t *values, int &val1, int &val2);
// Representative dot product functions
static __dpct_inline__ float vec_dot_q4_0_q8_1(const void * vbq,
const block_q8_1 * bq8_1, const int & iqs);
static __dpct_inline__ float vec_dot_q4_1_q8_1(const void * vbq,
const block_q8_1 * bq8_1, const int & iqs);
static __dpct_inline__ float vec_dot_q8_0_q8_1(const void * vbq,
const block_q8_1 * bq8_1, const int & iqs);
static __dpct_inline__ float vec_dot_q6_K_q8_1(const void * vbq,
const block_q8_1 * bq8_1, const int & iqs);
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| vbq | const void * | Yes | Pointer to a quantized weight block (type depends on quant format) |
| bq8_1 | const block_q8_1 * | Yes | Pointer to a q8_1-quantized input block |
| iqs | const int & | Yes | Quant sub-index within the block for partial dot product |
Outputs
| Name | Type | Description |
|---|---|---|
| return | float | Partial dot product result for the given quant sub-index |
Usage Examples
// Inside a MMVQ kernel, computing dot product for one block:
const block_q4_0 * x = (const block_q4_0 *) vx;
const block_q8_1 * y = (const block_q8_1 *) vy;
float partial = vec_dot_q4_0_q8_1(&x[ibx], &y[iby], iqs);
// Used as a template parameter:
mul_mat_vec_q<QK4_0, QI4_0, block_q4_0, VDR_Q4_0_Q8_1_MMVQ, vec_dot_q4_0_q8_1>
(vx, vy, dst, ncols, nrows, item_ct1);