Jump to content

Connect SuperML | Leeroopedia MCP: Equip your AI agents with best practices, code verification, and debugging knowledge. Powered by Leeroo — building Organizational Superintelligence. Contact us at founders@leeroo.com.

Implementation:Microsoft DeepSpeedExamples Tensor Parallel Utilities

From Leeroopedia


Knowledge Sources
Domains Data Generation, OpenAI API
Last Updated 2026-02-07 12:00 GMT

Overview

Provides utility functions for OpenAI API completion requests, JSON file I/O, and decoding argument configuration used to generate training data for tensor parallel fine-tuning.

Description

This module implements utilities for interacting with the OpenAI Completion API and managing JSON data files, primarily used for generating and processing training datasets. The OpenAIDecodingArguments dataclass defines configurable parameters for API requests including max_tokens, temperature, top_p, presence_penalty, frequency_penalty, and other sampling parameters.

The openai_completion function provides a robust batch completion interface that handles rate limiting with automatic retries and sleep, prompt length reduction when token limits are exceeded, batched prompt processing with configurable batch_size, and support for both single and multiple prompt inputs. It returns either raw OpenAIObject completions or extracted text depending on the return_text parameter, and groups results for multi-sample decoding (n > 1). The function supports both standard and chat model prompt formats.

The module also provides jdump and jload functions for JSON serialization and deserialization with automatic directory creation and support for both string paths and file-like objects. The _make_w_io_base and _make_r_io_base helper functions handle transparent file opening. The module reads the OPENAI_ORG environment variable at import time to configure the OpenAI organization context.

Usage

Use this module to generate synthetic training data via OpenAI API calls for subsequent fine-tuning with tensor parallelism, or to load/save JSON-formatted training datasets used by the tensor parallel training scripts.

Code Reference

Source Location

Signature

@dataclasses.dataclass
class OpenAIDecodingArguments(object):
    max_tokens: int = 1800
    temperature: float = 0.2
    top_p: float = 1.0
    n: int = 1
    stream: bool = False
    stop: Optional[Sequence[str]] = None
    presence_penalty: float = 0.0
    frequency_penalty: float = 0.0
    suffix: Optional[str] = None
    logprobs: Optional[int] = None
    echo: bool = False

def openai_completion(
    prompts: Union[str, Sequence[str], Sequence[dict[str, str]], dict[str, str]],
    decoding_args: OpenAIDecodingArguments,
    model_name="text-davinci-003",
    sleep_time=2,
    batch_size=1,
    max_instances=sys.maxsize,
    max_batches=sys.maxsize,
    return_text=False,
    **decoding_kwargs,
) -> Union[...]:
    ...

def jdump(obj, f, mode="w", indent=4, default=str):
    ...

def jload(f, mode="r"):
    ...

Import

from utils import openai_completion, OpenAIDecodingArguments, jdump, jload

I/O Contract

Inputs

Name Type Required Description
prompts str or list Yes A single prompt string, list of prompt strings, or list of chat message dicts
decoding_args OpenAIDecodingArguments Yes Dataclass with API decoding parameters (max_tokens, temperature, etc.)
model_name str No OpenAI model name (default: 'text-davinci-003')
sleep_time int No Seconds to sleep on rate-limit errors (default: 2)
batch_size int No Number of prompts per API request (default: 1)
max_instances int No Maximum number of prompts to process (default: sys.maxsize)
return_text bool No If True, return text strings instead of OpenAI objects (default: False)
f str or io.IOBase Yes File path or file object for jdump/jload operations

Outputs

Name Type Description
completions str or OpenAIObject or list API completion results; format depends on return_text and decoding_args.n
jload result dict or list Deserialized JSON data from file

Usage Examples

from utils import openai_completion, OpenAIDecodingArguments, jdump, jload

# Generate completions with the OpenAI API
decoding_args = OpenAIDecodingArguments(
    max_tokens=512,
    temperature=0.7,
    top_p=0.95
)
results = openai_completion(
    prompts=["Write a short story about AI.", "Explain quantum computing."],
    decoding_args=decoding_args,
    model_name="text-davinci-003",
    return_text=True,
    batch_size=2
)

# Save results to JSON
jdump(results, "completions.json")

# Load training data
data = jload("alpaca_data.json")
print(f"Loaded {len(data)} examples")

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment