Implementation:FMInference FlexLLMGen HF OPT Benchmark Runner

Knowledge Sources	FMInference_FlexLLMGen
Domains	LLM Inference, Benchmarking, Model Serving
Last Updated	2026-02-09 12:00 GMT

Overview

Benchmark runner that measures inference performance of OPT models using HuggingFace Accelerate or DeepSpeed ZeRO-Inference with support for CPU and disk offloading.

Description

This script provides an end-to-end benchmark harness for Meta's OPT family of language models (from 1.3B up to 175B parameters). It supports two execution backends: HuggingFace Accelerate (via load_checkpoint_and_dispatch with custom device maps) and DeepSpeed ZeRO Stage-3 (via deepspeed.initialize with ZeRO-Inference). The runner measures both prefill latency (processing the input prompt) and decode latency (autoregressive token generation), reporting throughput in tokens per second. It supports three memory placement strategies: GPU-only, CPU offload, and disk/NVMe offload, enabling benchmarking of models that exceed GPU memory capacity.

The script also supports dummy weights for benchmarking purposes (avoiding the need to download full model checkpoints), INT8 quantization via HuggingFace's load_in_8bit, and latency projection where a shorter generation run is extrapolated to estimate full-length decode latency.

Usage

Use this benchmark runner to compare FlexLLMGen's inference performance against baseline HuggingFace and DeepSpeed inference pipelines. It serves as the primary baseline measurement tool in the FlexLLMGen benchmark suite.

Code Reference

Source Location

• Repository: FMInference_FlexLLMGen
• File: benchmark/hf_ds/hf_opt.py
• Lines: 1-363

Signature

def run_generation(model_name, batch_size, prompt_len, gen_len, cut_gen_len,
                   cpu_offload, disk_offload, offload_dir, use_int8,
                   num_nodes, num_gpus_per_node, use_deepspeed, dummy,
                   output_file, pkl_file, no_log, verbose):

Import

# This is a standalone benchmark script, not imported as a module.
# Run via:
# HuggingFace mode:
#   python hf_opt.py --model facebook/opt-1.3b --batch-size 16 --cpu-offload
# DeepSpeed mode:
#   deepspeed --num_gpus 1 hf_opt.py --model facebook/opt-1.3b --batch-size 16 --use-deepspeed --cpu-offload

I/O Contract

Inputs

Name	Type	Required	Description
--model	str	No (default: facebook/opt-1.3b)	HuggingFace model identifier or path for the OPT model
--batch-size	int	No (default: 1)	Number of sequences to generate in parallel
--prompt-len	int	No (default: 512)	Length of input prompt tokens (padded)
--gen-len	int	No (default: 32)	Number of tokens to generate per sequence
--cut-gen-len	int	No	If set, generate this many tokens and project latency to gen-len
--cpu-offload	flag	No	Offload model weights to CPU memory
--disk-offload	flag	No	Offload model weights to NVMe disk storage
--offload-dir	str	No (default: ~/flexllmgen_offload_dir)	Directory for offloaded weight storage
--dummy	flag	No	Use randomly initialized weights for benchmarking
--int8	flag	No	Use INT8 quantization (HuggingFace mode only)
--num-gpus	int	No (default: 1)	Number of GPUs (HuggingFace mode)
--pin-memory	int	No (default: 1)	Enable pinned memory for CPU offloading (DeepSpeed mode)

Outputs

Name	Type	Description
log_file	text file	Benchmark log with prefill/decode latency, throughput, GPU peak memory, and model/cache sizes
stdout	text	Token outputs (verbose >= 2) and benchmark summary (verbose >= 1)

Usage Examples

# Benchmark OPT-1.3B with HuggingFace Accelerate + CPU offload
python benchmark/hf_ds/hf_opt.py \
    --model facebook/opt-1.3b \
    --batch-size 8 \
    --prompt-len 512 \
    --gen-len 32 \
    --cpu-offload

# Benchmark OPT-30B with DeepSpeed ZeRO-Inference + NVMe offload
deepspeed --num_gpus 1 benchmark/hf_ds/hf_opt.py \
    --model facebook/opt-30b \
    --batch-size 2 \
    --prompt-len 512 \
    --gen-len 32 \
    --disk-offload \
    --offload-dir /mnt/nvme/offload

# Benchmark with dummy weights for quick performance testing
python benchmark/hf_ds/hf_opt.py \
    --model facebook/opt-66b \
    --dummy \
    --batch-size 1 \
    --cpu-offload

Related Pages

• Principle:FMInference_FlexLLMGen_HuggingFace_DeepSpeed_Baseline_Inference