Principle:Togethercomputer Together python Batch Job Creation
| Attribute | Value |
|---|---|
| Type | Principle |
| Domains | Batch_Processing, Inference, API_Client |
| Repository | togethercomputer/together-python |
| Last Updated | 2026-02-15 16:00 GMT |
Overview
Mechanism for submitting a batch of inference requests as a single asynchronous job.
Description
Batch job creation submits an uploaded input file for offline batch processing. The job runs asynchronously with a 24-hour completion window. It processes each request in the input file against the specified endpoint and produces an output file with results.
The creation process takes two inputs:
- file_id -- The identifier of a previously uploaded JSONL file (with
purpose="batch-api"). - endpoint -- The target API endpoint string (e.g.,
/v1/chat/completionsor/v1/completions).
Upon submission, the server validates the input file, schedules the job, and returns a BatchJob object with status, progress tracking, and a unique job identifier. The job transitions through defined states: VALIDATING, IN_PROGRESS, and finally COMPLETED, FAILED, EXPIRED, or CANCELLED.
Usage
Use this principle after uploading a batch input file. The workflow is:
- Upload the JSONL input file with
purpose="batch-api"to obtain afile_id. - Call
Batches.create_batch()with thefile_idand the targetendpoint. - Store the returned batch job ID for subsequent status polling and result retrieval.
The completion window is fixed at 24 hours. Jobs that do not complete within this window transition to EXPIRED status.
Theoretical Basis
Asynchronous batch job creation follows the submit-and-poll pattern common in distributed processing systems:
- Submit: The client sends job parameters and receives an immediate acknowledgment with a job handle.
- Poll: The client periodically checks job status until completion.
- Retrieve: Once complete, the client downloads results using the output file identifier.
This pattern decouples job submission from execution, allowing the server to optimize scheduling, resource allocation, and throughput across many concurrent batch jobs. The 24-hour completion window provides a service-level agreement for job turnaround.
The BatchJob object tracks the following state model:
VALIDATING-- Server is validating the input file format and request schemas.IN_PROGRESS-- Requests are being processed.COMPLETED-- All requests finished successfully; output file is available.FAILED-- The job encountered an unrecoverable error.EXPIRED-- The job did not complete within the 24-hour window.CANCELLED-- The job was cancelled by the user.CANCELING-- Cancellation is in progress.