Principle:Bitsandbytes foundation Bitsandbytes Global Optimizer Configuration

Sources	Paper: 8-bit Optimizers via Block-wise Quantization, Repo: bitsandbytes
Domains	Optimization, Configuration
Last updated	2026-02-07 14:00 GMT

Overview

A global configuration mechanism that enables per-parameter optimizer hyperparameter overrides for mixed-precision optimization strategies. This allows different model parameters to use different optimizer settings (e.g., 8-bit vs. 32-bit precision, different learning rates) within a single optimizer instance.

Description

In practice, not all model parameters benefit equally from 8-bit optimization. Certain parameter types require higher precision:

• Embedding layers: These contain discrete lookup values where quantization error can cause vocabulary representation drift.
• Small tensors (bias vectors, layer norm parameters): With few elements, block-wise quantization has limited granularity and the memory savings are negligible.
• Critical layers: The first or last layers of a model may be more sensitive to precision loss.

The Global Optimizer Configuration pattern addresses this by providing a centralized manager where per-parameter optimizer overrides can be registered. The workflow is:

1. Register parameters with the global manager (must happen while parameters are still on CPU, before .cuda()).
2. Override configuration for specific parameters, setting per-parameter values for any optimizer hyperparameter (optim_bits, lr, betas, percentile_clipping, etc.).
3. Create the optimizer: During initialization, the optimizer queries the global manager and applies any registered overrides to the corresponding parameter configurations.

This enables mixed-precision optimization within a single optimizer:

• Large weight matrices use 8-bit Adam for maximum memory savings.
• Embedding layers use 32-bit Adam for precision-sensitive parameters.
• Specific layers can have different learning rates or beta values.

The overrides are stored in a dictionary keyed by the Python id() of each parameter tensor. During optimizer initialization, these are mapped to (group_index, param_index) pairs for efficient lookup during the optimization step.

Usage

The typical usage pattern involves three steps before the training loop:

import bitsandbytes as bnb

# 1. Get the global manager singleton
mng = bnb.optim.GlobalOptimManager.get_instance()

# 2. Register all model parameters (while still on CPU)
model = MyModel()
mng.register_parameters(model.parameters())

# 3. Override specific parameters
mng.override_config(model.embedding.weight, "optim_bits", 32)
mng.override_config(model.lm_head.weight, key_value_dict={"optim_bits": 32, "lr": 5e-4})

# Now move to GPU and create optimizer
model = model.cuda()
optimizer = bnb.optim.Adam8bit(model.parameters(), lr=1e-3)

This pattern is used when:

• Certain layers need different optimizer precision (8-bit vs. 32-bit)
• Per-layer learning rate schedules are desired
• Specific parameters require different gradient clipping thresholds

Theoretical Basis

The implementation follows the Singleton pattern:

• A single GlobalOptimManager instance exists per process, accessed via get_instance().
• The singleton maintains a mapping from parameter identity (id(tensor)) to configuration overrides.
• During register_parameters(), the manager records the group and parameter indices for each registered parameter.
• During optimizer initialization, get_config() checks whether any overrides exist for the current (group_index, param_index) pair and merges them into the default configuration.

The configuration resolution order is:

1. Start with the optimizer's default configuration (from param_groups and args).
2. Overlay any per-parameter overrides registered via GlobalOptimManager.

This ensures that unoverridden parameters use the optimizer's defaults, while specific parameters get their custom settings.

Related

• Implementation:Bitsandbytes_foundation_Bitsandbytes_GlobalOptimManager