Workflow:Intel Ipex llm Pipeline Parallel Inference
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Inference, Distributed_Computing |
| Last Updated | 2026-02-09 04:00 GMT |
Overview
End-to-end process for running Large Language Model inference across multiple Intel GPUs using IPEX-LLM's native pipeline parallelism to distribute model layers across devices.
Description
This workflow enables inference on models that are too large for a single GPU by splitting model layers across multiple Intel GPUs using pipeline parallelism. IPEX-LLM provides built-in pipeline parallel support via init_pipeline_parallel() and the pipeline_parallel_stages parameter in model loading. The model layers are automatically distributed across available GPUs, with each GPU processing its assigned layers and passing intermediate activations to the next GPU. This approach supports 13+ model families including Llama, Qwen, ChatGLM, Mistral, Baichuan, Yi, Phi, CodeLlama, Solar, and Vicuna. Combined with INT4/low-bit quantization, it enables running 13B-70B models on consumer Intel Arc GPUs.
Usage
Execute this workflow when you need to run inference on a model that exceeds the memory capacity of a single Intel GPU. Typical scenarios include running 13B models on two Intel Arc A770 GPUs (16GB each), or 70B models across four or more Intel Max GPUs. Requires multiple Intel GPUs accessible on the same machine.
Execution Steps
Step 1: Environment and Multi GPU Setup
Configure the Intel GPU runtime with oneAPI toolkit variables. Initialize the distributed process group required for pipeline parallelism. Verify that multiple Intel GPUs are visible and accessible. Set the number of pipeline parallel stages to match the available GPU count.
Key considerations:
- Requires torch.distributed initialization (oneCCL backend)
- Use mpirun to launch the script across GPU ranks
- Each GPU rank is assigned a portion of the model layers
- Set OMP_NUM_THREADS and related environment variables for optimal performance
Step 2: Pipeline Parallel Initialization
Call init_pipeline_parallel() from ipex_llm.transformers before loading the model. This registers the pipeline parallel hooks that will distribute model layers across GPUs during model loading. The initialization must occur before any model creation calls.
Key considerations:
- Must be called before AutoModelForCausalLM.from_pretrained()
- Sets up inter-GPU communication channels for activation transfer
- One-time initialization per process
Step 3: Model Loading with Layer Distribution
Load the model using IPEX-LLM's AutoModelForCausalLM.from_pretrained() with the pipeline_parallel_stages parameter set to the number of GPUs. The model layers are automatically partitioned and placed on their respective GPUs. Simultaneously apply low-bit quantization (sym_int4 default) to reduce per-layer memory.
Key considerations:
- pipeline_parallel_stages parameter controls number of GPU partitions
- Combined with load_in_low_bit for quantized pipeline parallel inference
- optimize_model=True enables IPEX-LLM's kernel optimizations
- use_cache=True is important for efficient autoregressive generation
- Falls back to AutoModel if AutoModelForCausalLM fails (for non-causal models)
Step 4: Tokenization and Input Preparation
Load the tokenizer and encode the input prompt. Each rank tokenizes the input independently, but only the last rank in the pipeline produces the final output tokens. Move input tensors to the XPU device corresponding to the local rank.
Key considerations:
- Input tensors must be on the correct XPU device (xpu:{local_rank})
- All ranks process the same input prompt
- Tokenizer loaded from the same model path as the model
Step 5: Inference and Output Collection
Run model.generate() with inference mode enabled. Perform a warmup generation pass to initialize caches and compile XPU kernels, then run the actual timed inference. Synchronize XPU devices after generation. Only the last rank in the pipeline (rank == gpu_num - 1) decodes and outputs the generated text.
Key considerations:
- Warmup pass is required for accurate timing measurements
- torch.xpu.synchronize() ensures all GPU operations complete before timing
- Only the last pipeline stage produces the final token logits
- First token latency and average rest token cost are tracked separately