Environment:Huggingface Transformers BitsAndBytes Quantization Env

Knowledge Sources	Huggingface Transformers import_utils.py
Domains	Quantization, Infrastructure, GPU
Last Updated	2026-02-13 20:00 GMT

Overview

Environment for 4-bit and 8-bit model quantization using the bitsandbytes library with CUDA GPU support.

Description

The bitsandbytes quantization environment enables loading and running large language models in reduced precision (4-bit NF4/FP4 or 8-bit LLM.int8) to drastically reduce GPU memory requirements. This environment requires the bitsandbytes library (>= 0.46.1), a CUDA-capable GPU, and PyTorch >= 2.4. The quantization is performed on-the-fly during model loading via the BitsAndBytesConfig.

Usage

Required for the Model Quantization workflow when using the bitsandbytes backend, and for QLoRA Fine-Tuning which combines 4-bit quantization with LoRA adapters. Use this when you need to load models that exceed your GPU's VRAM capacity.

System Requirements

Category	Requirement	Notes
OS	Linux (primary), Windows (experimental)	Best support on Linux
Hardware	NVIDIA GPU with CUDA	Compute capability >= 7.0 (Volta+)
VRAM	>= 6GB	For 7B model in 4-bit; scales with model size
CUDA	11.8 or 12.x	Must match bitsandbytes build

Dependencies

System Packages

NVIDIA CUDA Toolkit 11.8+ or 12.x

Python Packages

torch >= 2.4.0
bitsandbytes >= 0.46.1
accelerate >= 1.1.0
transformers >= 5.0

Credentials

HF_TOKEN: HuggingFace API token (required for gated models like Llama).

Quick Install

pip install transformers[torch] bitsandbytes>=0.46.1 accelerate>=1.1.0

Code Evidence

Minimum bitsandbytes version from src/transformers/utils/import_utils.py:97:

BITSANDBYTES_MIN_VERSION = "0.46.1"

Version-gated availability check from src/transformers/utils/import_utils.py:847-849:

@lru_cache
def is_bitsandbytes_available(min_version: str = BITSANDBYTES_MIN_VERSION) -> bool:
    is_available, bitsandbytes_version = _is_package_available("bitsandbytes", return_version=True)
    return is_available and version.parse(bitsandbytes_version) >= version.parse(min_version)

Common Errors

Error Message	Cause	Solution
`ImportError: bitsandbytes`	bitsandbytes not installed	`pip install bitsandbytes>=0.46.1`
`CUDA Setup failed`	CUDA not detected by bitsandbytes	Verify CUDA installation: `nvidia-smi`
`CUDA out of memory`	Still insufficient VRAM	Enable 4-bit instead of 8-bit, or reduce batch size
`ValueError: bnb quantization only supports GPU`	Attempting CPU quantization	BitsAndBytes requires a CUDA GPU

Compatibility Notes

AMD ROCm: bitsandbytes has experimental ROCm support. Requires bitsandbytes >= 0.48.3. Pre-quantized models may not work due to blocksize differences.
Intel XPU: Not supported by bitsandbytes.
Apple MPS: Not supported by bitsandbytes.
Windows: Experimental support; use WSL2 for best results.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment