Principle:Intel Ipex llm NPU Model Quantization

Knowledge Sources	Intel IPEX-LLM
Domains	Quantization, NPU, Model_Optimization
Last Updated	2026-02-09 04:00 GMT

Overview

Quantization technique for converting and running LLM models on Intel Neural Processing Units with low-bit weight representation.

Description

NPU model quantization converts standard floating-point model weights to low-bit integer representations (sym_int4, sym_int8) optimized for Intel NPU hardware. The IPEX-LLM NPU backend provides specialized from_pretrained, save_low_bit, and load_low_bit APIs that handle the quantization, NPU-specific compilation, and model serialization. The converted models achieve significant inference speedup on NPU hardware while maintaining acceptable accuracy through symmetric quantization.

Usage

Use this principle when deploying LLM models on Intel hardware with NPU accelerators (e.g., Intel Core Ultra processors). The quantized models run on the NPU's dedicated inference engine, freeing CPU and GPU for other tasks.

Theoretical Basis

Symmetric quantization maps float weights to integer range: $w_{int}=round(w_{float}/s)$ where $s$ is the scale factor.

For sym_int4: $w_{int}\in [-8,7]$, for sym_int8: $w_{int}\in [-128,127]$.

Pseudo-code Logic:

# Abstract NPU quantization flow
model = load_pretrained(model_path)
model_quantized = quantize(model, low_bit="sym_int4")
save_for_npu(model_quantized, output_path)

# Later: fast load
model = load_npu_model(output_path)
output = model.generate(input_ids)

Related Pages

• Implementation:Intel_Ipex_llm_NPU_Llama2_Inference
• Implementation:Intel_Ipex_llm_NPU_Model_Convert
• Implementation:Intel_Ipex_llm_NPU_Save_Load
• Implementation:Intel_Ipex_llm_NPU_BCE_Embedding
• Implementation:Intel_Ipex_llm_NPU_LLM_CLI_Cpp