Principle:Ggml org Llama cpp Importance Matrix Generation
| Field | Value |
|---|---|
| Principle Name | Importance Matrix Generation |
| Topic | Model Quantization |
| Workflow | Model_Quantization |
| Category | Calibration Data |
| Repository | Ggml_org_Llama_cpp |
Overview
Description
Importance matrix (imatrix) generation is the process of running a representative calibration dataset through a full-precision model to measure the relative importance of each weight element. The resulting importance matrix captures how much each weight contributes to the model's output, enabling importance-weighted quantization where more bits are allocated to critical weights and fewer bits to less impactful ones. This technique significantly improves the quality of aggressively quantized models (below 4 bits per weight) with minimal additional cost.
Usage
Importance matrix generation is an optional but highly recommended preprocessing step before quantization, especially when targeting low-bit quantization types such as IQ2_XXS, IQ2_XS, IQ3_XS, or IQ4_NL. The generated imatrix file is then passed as a parameter to the quantization tool. The workflow is:
- Run the full-precision model on calibration text to collect activation statistics
- Save the imatrix data to a GGUF file
- Supply the imatrix file when quantizing the model
Theoretical Basis
Why Uniform Quantization Is Suboptimal
Standard uniform quantization treats all weights equally, applying the same precision reduction across every element in a tensor. However, neural network weights are not equally important: some weights are on critical paths (e.g., attention key/value projections in early layers) while others have redundant or near-zero contributions. Uniform quantization wastes bits on unimportant weights and under-allocates bits to critical ones.
Activation-Based Importance Estimation
The importance of a weight w_{ij} in a matrix multiplication W * x is proportional to how much it contributes to the output variance. For a weight matrix W and input activations x, the importance of the j-th column of W is estimated by the squared activation sum:
importance_j = sum_over_samples( x_j^2 )
This quantity captures how frequently and how strongly each input dimension is activated across a representative dataset. Weights associated with highly activated input dimensions are more important because errors in those weights are amplified by larger input values.
Collection Mechanism
The imatrix collector intercepts matrix multiplication operations during forward passes. For each GGML_OP_MUL_MAT operation involving model weight tensors, it:
- Extracts the input activation tensor (src1)
- Computes the element-wise squared values of each activation vector
- Accumulates these squared values across all tokens and batches
- Tracks the count of observations per tensor for normalization
For models with Mixture-of-Experts (MoE) architectures, the collector handles GGML_OP_MUL_MAT_ID operations by tracking importance statistics per expert, using the expert routing IDs to attribute activations to the correct expert matrices.
Statistical Properties
The collected statistics enable several diagnostic analyses:
- Mean squared activation -- Average importance per weight column
- Entropy -- Measures how uniformly distributed the importance is; low entropy means a few weights dominate
- Active ratio -- Fraction of weight columns with non-negligible activation
- Cosine similarity -- Similarity between importance distributions of adjacent layers, useful for identifying layer pruning candidates
Calibration Data Requirements
The quality of the importance matrix depends on the calibration dataset being representative of the model's intended use. Key considerations:
- Diversity -- The dataset should cover the range of topics and styles the model will encounter
- Size -- Empirically, 100-200 chunks of context-length text provide stable importance estimates
- Domain matching -- For domain-specific models, calibration data should match the target domain