Implementation:Intel Ipex llm Model Generate PP

Knowledge Sources	IPEX-LLM
Domains	Distributed_Computing, NLP, Inference
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for generating text from a pipeline-parallel model with XPU synchronization and rank-based output collection.

Description

The model.generate() call on a pipeline-parallel model coordinates generation across all GPU ranks. Input must be on the correct XPU device. torch.xpu.synchronize() is required for accurate timing. Output is only meaningful on the last rank. The model provides first_token_time and rest_cost_mean attributes for latency analysis.

Usage

Use after loading a pipeline-parallel model. Always run a warmup generate call before benchmarking.

Code Reference

Source Location

Repository: IPEX-LLM
File: python/llm/example/GPU/Pipeline-Parallel-Inference/generate.py
Lines: 67-87

Signature

# Place input on correct device
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(f'xpu:{local_rank}')

# Generate tokens
output = model.generate(
    input_ids: torch.Tensor,
    max_new_tokens: int = 32
) -> torch.Tensor

# Synchronize for timing
torch.xpu.synchronize()

# Decode on last rank only
if local_rank == gpu_num - 1:
    output_str = tokenizer.decode(output[0], skip_special_tokens=True)

Import

import torch
from transformers import AutoTokenizer

I/O Contract

Inputs

Name	Type	Required	Description
input_ids	torch.Tensor	Yes	Tokenized prompt on xpu:{local_rank} device
max_new_tokens	int	No	Maximum tokens to generate (default 32)

Outputs

Name	Type	Description
output	torch.Tensor	Generated token IDs (meaningful only on last rank)
model.first_token_time	float	Time to generate first token (seconds)
model.rest_cost_mean	float	Average time per subsequent token (seconds)

Usage Examples

import torch
import time

local_rank = torch.distributed.get_rank()
gpu_num = 2

with torch.inference_mode():
    input_ids = tokenizer.encode("Once upon a time", return_tensors="pt").to(f'xpu:{local_rank}')

    # Warmup run
    _ = model.generate(input_ids, max_new_tokens=32)

    # Timed run
    st = time.time()
    output = model.generate(input_ids, max_new_tokens=32)
    torch.xpu.synchronize()
    end = time.time()

    output = output.cpu()
    if local_rank == gpu_num - 1:
        output_str = tokenizer.decode(output[0], skip_special_tokens=True)
        print(f'Inference time: {end-st} s')
        print(f"First token: {model.first_token_time:.4f}s, rest avg: {model.rest_cost_mean:.4f}s")
        print(output_str)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment