Implementation:Microsoft DeepSpeedExamples Tensor Parallel Utilities
| Knowledge Sources | |
|---|---|
| Domains | Data Generation, OpenAI API |
| Last Updated | 2026-02-07 12:00 GMT |
Overview
Provides utility functions for OpenAI API completion requests, JSON file I/O, and decoding argument configuration used to generate training data for tensor parallel fine-tuning.
Description
This module implements utilities for interacting with the OpenAI Completion API and managing JSON data files, primarily used for generating and processing training datasets. The OpenAIDecodingArguments dataclass defines configurable parameters for API requests including max_tokens, temperature, top_p, presence_penalty, frequency_penalty, and other sampling parameters.
The openai_completion function provides a robust batch completion interface that handles rate limiting with automatic retries and sleep, prompt length reduction when token limits are exceeded, batched prompt processing with configurable batch_size, and support for both single and multiple prompt inputs. It returns either raw OpenAIObject completions or extracted text depending on the return_text parameter, and groups results for multi-sample decoding (n > 1). The function supports both standard and chat model prompt formats.
The module also provides jdump and jload functions for JSON serialization and deserialization with automatic directory creation and support for both string paths and file-like objects. The _make_w_io_base and _make_r_io_base helper functions handle transparent file opening. The module reads the OPENAI_ORG environment variable at import time to configure the OpenAI organization context.
Usage
Use this module to generate synthetic training data via OpenAI API calls for subsequent fine-tuning with tensor parallelism, or to load/save JSON-formatted training datasets used by the tensor parallel training scripts.
Code Reference
Source Location
- Repository: Microsoft_DeepSpeedExamples
- File: training/tensor_parallel/utils.py
- Lines: 1-173
Signature
@dataclasses.dataclass
class OpenAIDecodingArguments(object):
max_tokens: int = 1800
temperature: float = 0.2
top_p: float = 1.0
n: int = 1
stream: bool = False
stop: Optional[Sequence[str]] = None
presence_penalty: float = 0.0
frequency_penalty: float = 0.0
suffix: Optional[str] = None
logprobs: Optional[int] = None
echo: bool = False
def openai_completion(
prompts: Union[str, Sequence[str], Sequence[dict[str, str]], dict[str, str]],
decoding_args: OpenAIDecodingArguments,
model_name="text-davinci-003",
sleep_time=2,
batch_size=1,
max_instances=sys.maxsize,
max_batches=sys.maxsize,
return_text=False,
**decoding_kwargs,
) -> Union[...]:
...
def jdump(obj, f, mode="w", indent=4, default=str):
...
def jload(f, mode="r"):
...
Import
from utils import openai_completion, OpenAIDecodingArguments, jdump, jload
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| prompts | str or list | Yes | A single prompt string, list of prompt strings, or list of chat message dicts |
| decoding_args | OpenAIDecodingArguments | Yes | Dataclass with API decoding parameters (max_tokens, temperature, etc.) |
| model_name | str | No | OpenAI model name (default: 'text-davinci-003') |
| sleep_time | int | No | Seconds to sleep on rate-limit errors (default: 2) |
| batch_size | int | No | Number of prompts per API request (default: 1) |
| max_instances | int | No | Maximum number of prompts to process (default: sys.maxsize) |
| return_text | bool | No | If True, return text strings instead of OpenAI objects (default: False) |
| f | str or io.IOBase | Yes | File path or file object for jdump/jload operations |
Outputs
| Name | Type | Description |
|---|---|---|
| completions | str or OpenAIObject or list | API completion results; format depends on return_text and decoding_args.n |
| jload result | dict or list | Deserialized JSON data from file |
Usage Examples
from utils import openai_completion, OpenAIDecodingArguments, jdump, jload
# Generate completions with the OpenAI API
decoding_args = OpenAIDecodingArguments(
max_tokens=512,
temperature=0.7,
top_p=0.95
)
results = openai_completion(
prompts=["Write a short story about AI.", "Explain quantum computing."],
decoding_args=decoding_args,
model_name="text-davinci-003",
return_text=True,
batch_size=2
)
# Save results to JSON
jdump(results, "completions.json")
# Load training data
data = jload("alpaca_data.json")
print(f"Loaded {len(data)} examples")