Principle:Bitsandbytes foundation Bitsandbytes Columnwise INT8 Quantization

Knowledge Sources	Bitsandbytes
Domains	Quantization, INT8, GPU_Optimization
Last Updated	2026-02-07 13:31 GMT

Overview

A fused GPU kernel technique that quantizes matrix columns to INT8 and transposes the result in a single pass, eliminating intermediate memory traffic.

Description

Standard columnwise quantization requires: (1) computing per-column statistics, (2) scaling and rounding to INT8, (3) transposing for subsequent matmul. Performing these as separate operations incurs multiple GPU memory reads/writes. This principle fuses all three into a single Triton kernel: each program instance processes one column, computes the column maximum, scales all values to [-127, 127], and writes the result directly in transposed layout. This reduces memory traffic by approximately 3x compared to separate operations.

Usage

Apply when a quantized transposed matrix is needed, such as in the backward pass of SwitchBack layers where the weight matrix must be columnwise-quantized and transposed for the input gradient computation.

Theoretical Basis

For a column j of matrix X with M rows:

${\text{absmax}}_j=\underset{i}{\max }|X_{ij}|$

$Q_{ji}=\text{round}(127\cdot X_{ij}/{\text{absmax}}_j)$

Note the output index is (j, i) — the transposition is achieved by writing to transposed memory locations.

Related Pages

• Implementation:Bitsandbytes_foundation_Bitsandbytes_Quantize_Columnwise_Transpose