Principle:Bitsandbytes foundation Bitsandbytes 4bit Dequantization And Matmul
Metadata
| Field | Value |
|---|---|
| Page Type | Principle |
| Knowledge Sources | Paper (QLoRA: Efficient Finetuning of Quantized LLMs), Repo (bitsandbytes) |
| Domains | Quantization, Linear_Algebra |
| Last Updated | 2026-02-07 14:00 GMT |
Overview
A fused computation pattern that dequantizes 4-bit weights and performs matrix multiplication in a single operation for efficient inference.
Description
During the forward pass of a 4-bit quantized linear layer, the stored 4-bit weights cannot be used directly for matrix multiplication. They must first be dequantized back to a higher-precision format (the compute dtype, typically bfloat16 or float16). The dequantization-and-matmul pattern combines this reconstruction step with the subsequent matrix multiplication into a coherent computation pipeline.
Dispatch Strategy
The computation follows different paths depending on the input shape:
- Single-token inference (GEMV path): When the input activation tensor has exactly one token (i.e.,
A.numel() == A.shape[-1]) and gradients are not required, an optimized gemv_4bit kernel is dispatched. This kernel performs the dequantization and matrix-vector multiplication in a fused GPU kernel, avoiding the materialization of the full dequantized weight matrix in memory. This path is significantly faster for autoregressive generation where tokens are processed one at a time.
- Batched inference and training (MatMul4Bit path): When the input contains multiple tokens or gradients are required, the MatMul4Bit autograd function is used. This function explicitly dequantizes the entire weight matrix via
dequantize_4bit(), transposes it, and then callstorch.nn.functional.linear()for the matrix multiplication. While this materializes the full-precision weight matrix temporarily, it enables standard batched GEMM operations and correct gradient computation.
- Alignment requirement: For the GEMV path, the hidden dimension of the input must be a multiple of the quantization block size. If this condition is not met, the computation falls back to the MatMul4Bit path with a warning.
Backward Pass
During backpropagation through a 4-bit layer (relevant for QLoRA fine-tuning), the backward pass of MatMul4Bit also performs dequantization. The gradient with respect to the input activations (grad_A) requires multiplying the output gradient by the dequantized weight matrix. The weights are dequantized again in the backward pass for this computation. Note that gradients with respect to the quantized weights themselves (grad_B) are not computed, as the quantized weights are frozen -- only LoRA adapter weights receive gradients.
Dtype Handling
The forward pass carefully manages dtypes:
- Input activations are cast to the compute dtype before multiplication.
- The bias (if present) is also cast to the compute dtype.
- Dequantized weights are cast to match the input dtype.
- The final output is cast back to the original input activation dtype before being returned.
This ensures that the 4-bit layer is a transparent replacement for a standard linear layer from the caller's perspective.
Usage
Dequantization-and-matmul happens automatically during the forward pass of any model that uses Linear4bit layers. Users do not call these functions directly. The pattern is active when:
- Running inference on a 4-bit quantized model.
- Running the forward pass during QLoRA fine-tuning.
- Any operation that triggers
model(input)orlayer(input)on a quantized model.
Theoretical Basis
Dequantization Algorithm
Dequantization reverses the quantization process to reconstruct approximate floating-point weights:
- Unpack: Extract the two 4-bit indices from each packed byte.
- Codebook lookup: Map each 4-bit index to its corresponding value in the NF4 or FP4 codebook. The codebook contains 16 floating-point values that represent the quantization levels.
- Rescale: Multiply each dequantized value by its block's absmax scaling factor to restore the original scale.
For double quantization (nested quantization), an additional step precedes the rescaling:
- Dequantize absmax: The 8-bit quantized absmax values are first dequantized using the second-level quantization state. The mean offset is added back to recover the approximate original absmax values.
- Rescale weights: The recovered absmax values are then used to rescale the codebook-looked-up weight values.
GEMV Optimization
The gemv_4bit kernel avoids materializing the full dequantized weight matrix. Instead, it performs the following in a single GPU kernel:
- Each GPU thread block loads a block of packed 4-bit weights and the corresponding absmax value.
- The thread block dequantizes the weights on-the-fly in registers.
- The partial dot product between the input vector and the dequantized weight block is computed.
- Partial results are reduced across thread blocks to produce the final output vector.
This approach saves significant memory bandwidth because the full float16 weight matrix (4x larger than the packed representation) is never written to or read from global GPU memory.
MatMul4Bit Autograd
The MatMul4Bit autograd function provides correct gradient computation for training:
- Forward:
output = linear(A, dequantize_4bit(B, state).T, bias) - Backward:
grad_A = grad_output @ dequantize_4bit(B, state)
The quantized weights B are saved for the backward pass. No gradient is computed for B itself (the quantized weights are frozen), only for the input activations A and optionally the bias.