Implementation:Turboderp org Exllamav2 Ext QMatrix

Knowledge Sources	Turboderp_org_Exllamav2
Domains	Quantization, CUDA, Linear_Algebra
Last Updated	2026-02-15 00:00 GMT

Overview

C++ extension for creating, managing, and performing GEMM operations on quantized weight matrices in both ExLLaMA and GPTQ formats, with tensor-parallel support.

Description

ext_qmatrix.cpp provides the interface between Python and the GPU-resident quantized matrix objects (QMatrix). It supports two quantization formats:

ExLLaMA format -- Uses q_weight, q_perm, q_invperm, q_scale, q_scale_max, q_groups, and q_group_map tensors. The weight data is stored as packed 32-bit integers with per-group scaling and optional column permutation for better quantization quality.

GPTQ format -- Uses gptq_qzeros, gptq_scales, and gptq_g_idx tensors alongside q_weight. Detected when ExLLaMA scale tensors are on the meta device.

The principal functions are:

make_q_matrix -- Constructs a QMatrix object on the GPU. Validates tensor dtypes (q_weight as kInt, scales as kHalf, etc.), determines height from either group_map or weight shape, allocates a dequantization buffer (temp_dq with max_dq_rows), and returns an opaque handle. Throws on CUDA OOM.

make_q_matrix_split -- Variant for tensor-parallel weight splitting. Only supported for ExLLaMA format (GPTQ raises an error). Passes split=true to the QMatrix constructor to indicate the matrix represents a shard of a larger weight.

gemm_half_q_half -- Performs matrix multiplication C = A @ Q_matrix where A and C are FP16 dense tensors and Q_matrix is the quantized weight. Dispatches to gemm_half_q_half_cuda with cuBLAS handle and stream.

gemm_half_q_half_tp -- Tensor-parallel variant that iterates over device-local tensor/QMatrix pairs, executing GEMM on each device using the TP context's streams.

reconstruct -- Fully dequantizes a QMatrix back to a dense FP16 tensor. Useful for debugging or mixed-precision fallback.

matrix_q4_to_fp16 / matrix_fp16_to_q4 -- Convert between 4-bit packed (kByte) and FP16 representations with per-group scales.

make_group_map -- Builds a group mapping tensor from q_groups metadata, computing per-row group index and remaining rows in each group. Returns a torch.ShortTensor.

Usage

Use make_q_matrix during model loading to wrap quantized weight tensors into GPU-optimized matrix objects. Use gemm_half_q_half for all linear layer forward passes with quantized weights. Use reconstruct when you need to inspect dequantized weights. Use make_group_map during model preparation to precompute the group layout.

Code Reference

Source Location

Repository: Turboderp_org_Exllamav2
File: exllamav2/exllamav2_ext/ext_qmatrix.cpp
Lines: 1-361

Signature

uintptr_t make_q_matrix(
    torch::Tensor q_weight,       // kInt, packed quantized weights
    torch::Tensor q_perm,         // kShort, column permutation
    torch::Tensor q_invperm,      // kShort, inverse permutation
    torch::Tensor q_scale,        // kInt, per-group scales (ExLLaMA)
    torch::Tensor q_scale_max,    // kHalf, scale maxima
    torch::Tensor q_groups,       // kShort, group boundaries
    torch::Tensor q_group_map,    // kShort, row-to-group mapping
    torch::Tensor gptq_qzeros,   // kInt, GPTQ zero points
    torch::Tensor gptq_scales,   // kHalf, GPTQ scales
    torch::Tensor gptq_g_idx,    // kInt, GPTQ group index
    torch::Tensor bias,           // kHalf, optional bias
    torch::Tensor temp_dq,        // kHalf, dequantization buffer
    int max_dq_rows
);

uintptr_t make_q_matrix_split(
    /* same parameters as make_q_matrix */
);

void gemm_half_q_half(
    torch::Tensor a,      // [m, k] FP16 input
    uintptr_t b,           // QMatrix handle
    torch::Tensor c,      // [m, n] FP16 output
    bool force_cuda        // force CUDA kernel over cuBLAS
);

void gemm_half_q_half_tp(
    const std::vector<torch::Tensor> &a,
    const std::vector<uintptr_t> &b,
    const std::vector<torch::Tensor> &c,
    bool force_cuda,
    uintptr_t tp_context,
    int t_device
);

void reconstruct(
    uintptr_t q_handle,
    torch::Tensor output   // [height, width] FP16 output
);

void matrix_q4_to_fp16(torch::Tensor in, torch::Tensor scales, torch::Tensor out);
void matrix_fp16_to_q4(torch::Tensor in, torch::Tensor out, torch::Tensor scales);
torch::Tensor make_group_map(torch::Tensor& q_groups, int num_qrows);

Import

from exllamav2.ext import exllamav2_ext as ext_c

I/O Contract

Inputs

Parameter	Type	Description
q_weight	`torch.Tensor` (kInt)	Packed quantized weight data
q_perm / q_invperm	`torch.Tensor` (kShort)	Column permutation and its inverse (ExLLaMA format)
q_scale	`torch.Tensor` (kInt)	Per-group scale factors (ExLLaMA format)
gptq_qzeros / gptq_scales	`torch.Tensor`	GPTQ zero points (kInt) and scales (kHalf)
bias	`torch.Tensor` (kHalf)	Optional bias vector; meta device if unused
temp_dq	`torch.Tensor` (kHalf)	Temporary buffer for partial dequantization, size >= width * min(max_dq_rows, height)
a	`torch.Tensor` (kHalf)	Input activation matrix [m, k]
force_cuda	bool	If true, forces the custom CUDA GEMM kernel instead of cuBLAS

Outputs

Function	Return	Description
make_q_matrix	`uintptr_t`	Opaque handle to GPU-resident QMatrix object
make_q_matrix_split	`uintptr_t`	Handle to a split QMatrix shard for tensor parallelism
gemm_half_q_half	void	Writes result of A @ Q into tensor c
reconstruct	void	Writes fully dequantized FP16 matrix into output
matrix_q4_to_fp16	void	Writes dequantized FP16 values into out tensor
matrix_fp16_to_q4	void	Writes 4-bit packed values and scales into out/scales tensors
make_group_map	`torch.Tensor` (kShort)	Group mapping tensor of shape [2 * total_rows]

Usage Examples

from exllamav2.ext import exllamav2_ext as ext_c

# Create quantized matrix from ExLLaMA format weights
q_handle = ext_c.make_q_matrix(
    q_weight, q_perm, q_invperm, q_scale, q_scale_max,
    q_groups, q_group_map,
    gptq_qzeros, gptq_scales, gptq_g_idx,
    bias, temp_dq, max_dq_rows=128
)

# Perform quantized GEMM: output = input @ quantized_weight
ext_c.gemm_half_q_half(input_fp16, q_handle, output_fp16, force_cuda=False)

# Dequantize for inspection
ext_c.reconstruct(q_handle, full_weight_fp16)

# Build group map during model setup
group_map = ext_c.make_group_map(q_groups, num_qrows)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment