Implementation:Sgl project Sglang GGUF Quantization

Knowledge Sources	Sgl_project_Sglang
Domains	Kernel, Quantization, GGUF
Last Updated	2026-02-10 00:00 GMT

Overview

Python interface for GGUF/GGML quantized model operations including dequantization, matrix multiplication, and Mixture-of-Experts support.

Description

The quantization/gguf.py module provides six functions for working with GGML-quantized model weights. ggml_dequantize converts GGML-quantized weight tensors to a target dtype given the quantization type identifier and output shape (M x N). ggml_mul_mat_vec_a8 performs optimized matrix-vector multiplication between GGML-quantized weights and an input vector, suitable for single-token decode. ggml_mul_mat_a8 performs full matrix-matrix multiplication for batched inputs (prefill). For MoE models, ggml_moe_a8 handles routed matrix multiplication using sorted token routing information (sorted_token_ids, expert_ids, num_token_post_padded), while ggml_moe_a8_vec provides a vectorized MoE path using topk_ids directly. ggml_moe_get_block_size returns the quantization block size for a given GGML quantization type. All functions delegate to torch.ops.sgl_kernel.* C++ ops.

Usage

Use these functions for inference with llama.cpp-compatible GGUF quantized models within SGLang, supporting the full range of GGML quantization types (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, etc.) for both dense and MoE architectures.

Code Reference

Source Location

• Repository: Sgl_project_Sglang
• File: sgl-kernel/python/sgl_kernel/quantization/gguf.py
• Lines: 1-63

Signature

def ggml_dequantize(
    weight: torch.Tensor, quant_type: int,
    M: int, N: int, dtype: torch.dtype,
) -> torch.Tensor: ...

def ggml_mul_mat_vec_a8(
    weight: torch.Tensor, x: torch.Tensor,
    quant_type: int, row: int,
) -> torch.Tensor: ...

def ggml_mul_mat_a8(
    weight: torch.Tensor, x: torch.Tensor,
    quant_type: int, row: int,
) -> torch.Tensor: ...

def ggml_moe_a8(
    input: torch.Tensor, weight: torch.Tensor,
    sorted_token_ids: torch.Tensor, expert_ids: torch.Tensor,
    num_token_post_padded: torch.Tensor,
    type: int, row: int, topk: int, tokens: int,
) -> torch.Tensor: ...

def ggml_moe_a8_vec(
    input: torch.Tensor, weight: torch.Tensor,
    topk_ids: torch.Tensor, top_k: int,
    type: int, row: int, tokens: int,
) -> torch.Tensor: ...

def ggml_moe_get_block_size(type: int) -> int: ...

Import

from sgl_kernel.quantization.gguf import (
    ggml_dequantize,
    ggml_mul_mat_vec_a8,
    ggml_mul_mat_a8,
    ggml_moe_a8,
    ggml_moe_a8_vec,
    ggml_moe_get_block_size,
)
# or via top-level
from sgl_kernel import ggml_dequantize, ggml_mul_mat_a8

I/O Contract

Inputs

Name	Type	Required	Description
weight	torch.Tensor	Yes	GGML-quantized weight tensor
x	torch.Tensor	Yes (matmul)	Input activation tensor
quant_type	int	Yes	GGML quantization type identifier (e.g., Q4_0, Q8_0)
M	int	Yes (dequant)	Number of output rows
N	int	Yes (dequant)	Number of output columns
dtype	torch.dtype	Yes (dequant)	Target dtype for dequantized output
row	int	Yes	Number of rows in the weight matrix
input	torch.Tensor	Yes (MoE)	Input activation for MoE GEMM
sorted_token_ids	torch.Tensor	Yes (moe_a8)	Token IDs sorted by expert assignment
expert_ids	torch.Tensor	Yes (moe_a8)	Expert IDs per token group
num_token_post_padded	torch.Tensor	Yes (moe_a8)	Token count after padding
topk_ids	torch.Tensor	Yes (moe_a8_vec)	Top-k expert IDs per token
top_k	int	Yes (MoE)	Number of experts per token
tokens	int	Yes (MoE)	Number of input tokens
type	int	Yes	GGML quantization type for MoE/block_size functions

Outputs

Name	Type	Description
result	torch.Tensor	Dequantized weight tensor (ggml_dequantize)
result	torch.Tensor	Matrix multiplication output (ggml_mul_mat_vec_a8, ggml_mul_mat_a8)
result	torch.Tensor	MoE GEMM output (ggml_moe_a8, ggml_moe_a8_vec)
block_size	int	Quantization block size (ggml_moe_get_block_size)

Usage Examples

from sgl_kernel.quantization.gguf import (
    ggml_dequantize, ggml_mul_mat_a8, ggml_moe_get_block_size
)

# Dequantize GGML weights
dequantized = ggml_dequantize(
    weight_tensor, quant_type=2, M=4096, N=4096,
    dtype=torch.float16
)

# Matrix multiplication with GGML-quantized weights
output = ggml_mul_mat_a8(weight_tensor, input_tensor, quant_type=2, row=4096)

# Get block size for a quantization type
block_size = ggml_moe_get_block_size(type=2)

Related Pages

• Environment:Sgl_project_Sglang_CUDA_GPU_Runtime