Workflow:FMInference FlexLLMGen Text Completion API
| Knowledge Sources | |
|---|---|
| Domains | LLM_Inference, Text_Generation, API_Usage |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
End-to-end process for running interactive text completion with OPT models using FlexLLMGen's generation API, demonstrating programmatic usage for Q&A and information extraction tasks.
Description
This workflow demonstrates how to use FlexLLMGen as a Python library for text completion tasks. It covers constructing prompts, configuring the offloading policy, initializing the model, calling the generation API, and decoding outputs. The workflow follows the HuggingFace-style generation interface with support for sampling parameters, batched inputs, and stop tokens. This is the recommended starting point for developers integrating FlexLLMGen into their own applications.
Usage
Execute this workflow when you want to programmatically generate text completions using OPT models through the FlexLLMGen API, rather than using the command-line benchmark interface. Suitable for building applications such as Q&A systems, information extraction pipelines, or custom inference services on hardware with limited GPU memory.
Execution Steps
Step 1: Prepare Input Prompts
Construct the text prompts for the completion task. Prompts should be formatted as plain text strings with appropriate structure for the intended task (e.g., few-shot Q&A format, instruction format, or extraction format). Multiple prompts can be batched together for higher throughput.
Key considerations:
- Prompts are provided as a list of strings
- Batching more prompts improves throughput due to FlexLLMGen's block scheduling
- The tokenizer handles padding to uniform length (padding_side="left")
Step 2: Initialize Execution Environment
Create the FlexLLMGen execution environment with the offloading directory. This sets up the GPU, CPU, and disk devices with their associated copy threads and CUDA streams for overlapped I/O.
Key considerations:
- Call ExecutionEnv.create(offload_dir) with the path to the offload directory
- The offload directory is used for disk-tier tensor storage
Step 3: Configure Offloading Policy
Create a Policy object specifying the offloading percentages for weights, KV cache, and activations across GPU, CPU, and disk. Also configure compression settings if memory is limited. The Policy takes the batch size as input, which must match the number of prompts.
Key considerations:
- gpu_batch_size should match the number of input prompts
- num_gpu_batches defaults to 1 for the API usage pattern
- Enable overlap and sep_layer for best performance
- Compression uses 4-bit group-wise quantization with group_size=64
Step 4: Load Tokenizer and Model
Initialize the HuggingFace AutoTokenizer for the OPT model family and the OptLM model with the configured policy. The tokenizer handles text-to-token conversion and the model loads weights distributed across the memory hierarchy.
Key considerations:
- The tokenizer is initialized from the OPT model name (e.g., "facebook/opt-30b")
- Set padding_side="left" and add_bos_token=False for correct OPT behavior
- Define a stop token (e.g., newline) to control generation boundaries
- OptLM accepts the model name or an OptConfig object, the environment, weight path, and policy
Step 5: Tokenize and Generate
Tokenize the prompts with padding to uniform length, then call the model's generate method with sampling parameters. The generate method runs the full prefill and autoregressive generation loop using the block schedule.
Key considerations:
- Use tokenizer(prompts, padding="max_length", max_length=N) for uniform input lengths
- The generate API supports do_sample, temperature, max_new_tokens, and stop parameters
- Output is a numpy array of token IDs including both prompt and generated tokens
Step 6: Decode Outputs and Shutdown
Decode the output token IDs back to text using the tokenizer's batch_decode method, then close the execution environment to release resources.
Key considerations:
- Use skip_special_tokens=True in batch_decode for clean output
- Always call env.close_copy_threads() to properly release background I/O threads