Workflow:Togethercomputer Together python Batch Inference
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Inference, Batch_Processing |
| Last Updated | 2026-02-15 16:00 GMT |
Overview
End-to-end process for submitting large-scale inference jobs to Together AI's batch processing API, from input file preparation through job creation, monitoring, and result retrieval.
Description
This workflow covers the batch inference pipeline for processing large volumes of requests with a 24-hour turnaround time. It uses the Together batch API to submit a file containing multiple inference requests (chat completions or other endpoints), process them asynchronously on Together's infrastructure, and retrieve the completed results. Batch inference is cost-effective for non-real-time workloads such as dataset evaluation, bulk content generation, or offline analysis.
Usage
Execute this workflow when you have a large number of inference requests that do not require real-time responses. This applies to evaluation benchmarks, dataset augmentation, bulk classification, or any scenario where throughput is more important than latency. The batch API processes requests within a 24-hour window at reduced cost compared to synchronous API calls.
Execution Steps
Step 1: Input File Preparation
Prepare a JSONL file where each line contains a complete API request body. Each request follows the same format as the corresponding synchronous API endpoint (e.g., chat completions). Each line should include a custom_id for correlating inputs with outputs.
Key considerations:
- Each line is a standalone JSON object representing one API request
- The format matches the target endpoint's request schema
- Include custom_id fields to match results back to inputs
- The file must be valid JSONL with no trailing commas or malformed lines
Step 2: File Upload
Upload the batch input file to Together's storage using the files API with purpose set to "batch-api". The upload returns a file ID that uniquely identifies the input data for the batch job.
Key considerations:
- Set purpose="batch-api" to distinguish batch input files from fine-tuning data
- The same upload routing applies: single upload for small files, multipart for large files
- The returned file ID is required for creating the batch job
Step 3: Batch Job Creation
Create a batch job by specifying the uploaded file ID and the target API endpoint path. The batch API schedules the job for processing and returns a batch job object with a unique identifier for monitoring.
Key considerations:
- The endpoint parameter specifies which API to call (e.g., "/v1/chat/completions")
- The job enters a queue and may not start immediately
- The returned batch job ID is used for all subsequent status checks
Step 4: Job Monitoring
Poll the batch job status periodically to check progress. The job transitions through states from queued to processing to completed (or failed). The batch object includes metadata about total requests, completed count, and error count.
Key considerations:
- Use get_batch() with the batch job ID to check current status
- Status values include queued, processing, completed, failed, and cancelled
- list_batches() returns all batch jobs for the account
- Jobs can be cancelled with cancel_batch() if no longer needed
Step 5: Result Retrieval
Once the batch job completes, download the output file containing all inference results. The output file ID is available in the completed batch job's metadata. Each line in the output corresponds to a request from the input file, matched by custom_id.
Key considerations:
- The output_file_id field on the completed batch job points to the results file
- Use files.retrieve_content() with the output file ID to download results
- The output parameter specifies the local file path for saving
- Each output line contains the original custom_id for correlation with inputs