Implementation:Turboderp org Exllamav2 FPx Quantization

Knowledge Sources	Turboderp_org_Exllamav2
Domains	Quantization, Experimental
Last Updated	2026-02-15 00:00 GMT

Overview

Python module providing sub-byte floating-point (FPx) quantization utilities for converting tensors between FP32 and arbitrary low-bit floating-point formats with configurable exponent and mantissa bit widths.

Description

fpx.py implements bidirectional conversion between FP32 and arbitrary sub-byte floating-point formats (e.g., FP4_E2M1, FP6_E3M2, FP6_E2M3). The code is adapted from the Aphrodite engine's FP6 utilities.

Core conversion functions:

_f32_to_fpx_unpacked(x, ebits, mbits) converts FP32 tensors to sub-byte FPx format stored as uint8. The algorithm handles three branches:

• Saturated values: Inputs exceeding the FPx maximum are clamped to max_int.
• Denormal values: Small inputs below min_normal are converted using a magic-number denormalization trick with additive bias.
• Normal values: Standard values undergo exponent bias adjustment and round-to-nearest-even using a mantissa oddness correction and magic adder.

Signs are extracted, stripped during processing, and reattached at the end. The constraint 1 + ebits + mbits <= 8 ensures the result fits in uint8.

_fpx_unpacked_to_f32(x, ebits, mbits) converts uint8-encoded FPx values back to FP32. It handles:

• Zero values: Detected and set directly.
• Denormal values: Reconstructed by iterating over all possible mantissa values, left-shifting until overflow to create an implicit leading 1, with a fast path for mbits==1.
• Normal values: Exponent bias is converted from FPx to FP32 range and mantissa bits are shifted to FP32 positions.

Higher-level functions:

to_scaled_tc_fpx(tensor, ebits, mbits) quantizes a tensor with per-row scaling: computes the absolute maximum per row, divides by the FPx max_normal to get a scale factor, then quantizes the normalized tensor. Returns a tuple of (quantized uint8 tensor, FP16 scale).

from_scaled_tc_fpx(fpx_unpacked, ebits, mbits, scale) dequantizes by converting FPx to FP32 and multiplying by the scale factor.

fpxify(tensor, exponent, mantissa) is a convenience function that performs a round-trip quantize-dequantize on a tensor, useful for testing quantization error. It moves the tensor to CUDA, quantizes, dequantizes, and returns the result in FP16.

Usage

These utilities are used in the experimental quantization pipeline for evaluating sub-byte floating-point formats. They allow researchers to test different exponent/mantissa configurations and measure the impact on model quality before committing to a specific quantization format.

Code Reference

Source Location

• Repository: Turboderp_org_Exllamav2
• File: exllamav2/experimental/fpx.py
• Lines: 1-249

Signature

def _f32_to_fpx_unpacked(x: Tensor, ebits: int, mbits: int) -> Tensor:
    ...

def _fpx_unpacked_to_f32(x: Tensor, ebits: int, mbits: int) -> Tensor:
    ...

def to_scaled_tc_fpx(tensor: Tensor, ebits: int, mbits: int) -> tuple[Tensor, Tensor]:
    ...

def from_scaled_tc_fpx(fpx_unpacked: Tensor, ebits: int, mbits: int, scale=None) -> Tensor:
    ...

def fpxify(tensor: torch.Tensor, exponent: int, mantissa: int) -> torch.Tensor:
    ...

Import

from exllamav2.experimental.fpx import to_scaled_tc_fpx, from_scaled_tc_fpx, fpxify

I/O Contract

Function	Input	Output	Description
_f32_to_fpx_unpacked	x (Tensor, float32), ebits (int), mbits (int)	Tensor (uint8)	FP32 to FPx; bits stored in least significant bits of uint8
_fpx_unpacked_to_f32	x (Tensor, uint8), ebits (int), mbits (int)	Tensor (float32)	FPx to FP32 dequantization
to_scaled_tc_fpx	tensor (Tensor), ebits (int), mbits (int)	(Tensor uint8, Tensor FP16 scale)	Per-row scaled quantization; scale = row_absmax / max_normal
from_scaled_tc_fpx	fpx_unpacked (Tensor uint8), ebits, mbits, scale (optional)	Tensor (float32)	Dequantize and rescale
fpxify	tensor (Tensor), exponent (int), mantissa (int)	Tensor (FP16)	Round-trip quantize-dequantize for testing

Constraint	Description
Bit width	1 + ebits + mbits <= 8 (must fit in uint8)
Input dtype	_f32_to_fpx_unpacked requires float32; _fpx_unpacked_to_f32 requires uint8
Special values	No NaN or Inf support; out-of-range values are clamped to FPx max

Usage Examples

import torch
from exllamav2.experimental.fpx import to_scaled_tc_fpx, from_scaled_tc_fpx, fpxify

# Quantize a weight tensor to FP6 (E3M2)
weight = torch.randn(4096, 4096, dtype=torch.float32, device="cuda")
quantized, scale = to_scaled_tc_fpx(weight, ebits=3, mbits=2)
# quantized: shape (4096, 4096), dtype uint8
# scale: shape (4096,), dtype float16

# Dequantize back to FP32
reconstructed = from_scaled_tc_fpx(quantized, ebits=3, mbits=2, scale=scale)
# reconstructed: shape (4096, 4096), dtype float32

# Quick round-trip test with FP4 (E2M1)
original = torch.randn(1024, 1024, dtype=torch.float16, device="cuda")
roundtripped = fpxify(original, exponent=2, mantissa=1)
error = (original.float() - roundtripped.float()).abs().mean()
print(f"Mean absolute error: {error:.6f}")

Related Pages

• Turboderp_org_Exllamav2_Ext_Hadamard -- Hadamard matrices used in quantization-aware rotations
• Turboderp_org_Exllamav2_Model_Diff -- Model comparison tool for evaluating quantization quality