Implementation:Ggml org Ggml Cpu spacemit ime1 kernels

Metadata

Field	Value
Page Type	Implementation (SpacemiT IME1 Kernels)
Knowledge Sources	GGML
Domains	ML_Infrastructure, Tensor_Computing, CPU_Backend, Quantized_Matrix_Multiplication
Last Updated	2025-05-15 12:00 GMT

Overview

Implements the low-level SpacemiT IME1 computation kernels for int8-by-int4 quantized GEMM and activation quantization using RISC-V vector assembly.

Description

spacemit/ime1_kernels.cpp provides the performance-critical inner kernels for SpacemiT IME1 hardware acceleration, implementing the actual vectorized quantization and matrix multiplication at the assembly level. Key components include:

1. Activation quantization (quantize_a_4row_i8): Quantizes 4 rows of float activations simultaneously to int8 with block-wise scaling using RISC-V vector assembly. Key RVV instructions used:
   • vfabs.v -- Absolute values for scale calculation.
   • vfredmax.vs -- Block maximum for scale factor.
   • vfmul.vf -- Scale application.
   • vfcvt.x.f.v -- Float-to-integer conversion.
   • vnclip.wx -- Narrowing clip to int8.
   • Supports block lengths of 16, 32, 64, and 128.
2. Assembly macros:
   • QUANTIZEM4ROW_KERNEL -- Core quantization sequence using v-registers: compute abs max, derive scale, multiply, convert, and narrow.
   • QUANTIZEM4ROW_STORE -- Stores the narrowed int8 results in strided interleaved layout.
3. GEMM kernel (gemm_kernel_i8i4): Performs the int8-by-int4 matrix multiplication using the SpacemiT IME1 hardware instructions. Handles variable row counts with a return value indicating how many rows were processed.
4. Namespace: All functions reside in sqnbitgemm_spacemit_ime::ime1.

Usage

These kernels are called by ime.cpp for the inner computation loops. They are not intended for direct use outside the SpacemiT backend.

Code Reference

Source Location

GGML repo, file: src/ggml-cpu/spacemit/ime1_kernels.cpp (3196 lines).

Signature

namespace sqnbitgemm_spacemit_ime::ime1 {

// Quantize 4 rows of float activations to int8
void quantize_a_4row_i8(
    size_t BlkLen,
    const float * A,
    size_t CountK,
    std::byte * QuantA);

// Perform int8-by-int4 GEMM using IME1 hardware
int32_t gemm_kernel_i8i4(
    size_t blk_len,
    const std::byte * a_row,
    const std::byte * b_col,
    const void * b_col_scale,
    const std::byte * b_col_zp,
    float * c_blk,
    int32_t rows_remaining,
    size_t count_n,
    size_t gemm_k,
    size_t k_blks,
    size_t ldc,
    const float * bias,
    size_t scale_stride);

} // namespace sqnbitgemm_spacemit_ime::ime1

Import

#include "ime_kernels.h"

I/O Contract

Inputs

Parameter	Type	Required	Description
BlkLen	size_t	Yes	Block length for quantization (16, 32, 64, or 128).
A	const float *	Yes	Float activation matrix (4 rows of CountK elements).
CountK	size_t	Yes	Number of elements per row.
a_row	const std::byte *	Yes (GEMM)	Quantized int8 activation data.
b_col	const std::byte *	Yes (GEMM)	Packed int4 weight data with interleaved scales.
rows_remaining	int32_t	Yes (GEMM)	Number of rows to process in this kernel call.

Outputs

Output	Type	Description
QuantA	std::byte *	Quantized int8 activation data with interleaved scale factors.
c_blk	float *	GEMM result block in f32 format.
Return value	int32_t	Number of rows actually processed by the kernel.

Usage Examples

Activation Quantization (Internal)

// Called from ime.cpp during the GEMM pipeline:
namespace ime = sqnbitgemm_spacemit_ime::ime1;

// Quantize 4 rows of activations to int8
std::byte quant_buffer[4 * lda];
ime::quantize_a_4row_i8(32, float_activations, k, quant_buffer);

// Execute int8-by-int4 GEMM
int32_t rows_done = ime::gemm_kernel_i8i4(
    32,            // block length
    quant_buffer,  // quantized activations
    packed_weights, nullptr, nullptr,
    output,        // float output
    4,             // rows remaining
    n_count, k, k_blks, ldc, nullptr, sizeof(uint16_t));

Related Pages

• Ggml_org_Ggml_Cpu_spacemit_ime -- The backend integration that dispatches to these kernels.
• Ggml_org_Ggml_Cpu_quantization -- Quantization primitives complementing these IME-specific formats.
• Ggml_org_Ggml_Cpu_kleidiai_kernels -- ARM KleidiAI: analogous micro-kernel wrappers for ARM.