Heuristic:Togethercomputer Together python Multipart Upload Strategy
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, File_Upload |
| Last Updated | 2026-02-15 16:00 GMT |
Overview
Optimal file upload strategy using multipart uploads for files >= 5GB, with sliding-window concurrency and S3-compliant part sizing.
Description
The Together SDK automatically switches between single-request upload and multipart upload based on file size. For files >= 5GB, the SDK splits the file into parts (target 250MB each, minimum 5MB per S3 requirements) and uploads up to 4 parts concurrently using a sliding window pattern. This balances throughput with memory usage and respects S3 backend constraints.
Usage
This heuristic applies automatically when uploading files via `client.files.upload()`. Understanding the thresholds and limits is useful when:
- Uploading large fine-tuning datasets (up to 50.1 GB)
- Debugging upload timeouts or failures
- Planning upload bandwidth requirements
The Insight (Rule of Thumb)
- Action: Files < 5GB upload in a single request. Files >= 5GB use multipart upload automatically.
- Value: Target part size: 250MB. Minimum part: 5MB (S3 requirement). Maximum parts: 250. Max concurrent: 4.
- Trade-off: Multipart upload adds overhead for small files but enables reliable transfer of large datasets. Each part has a 5-minute (300s) timeout.
Upload thresholds:
- < 5 GB: Single-request upload
- >= 5 GB and <= 50.1 GB: Multipart upload with optimal part calculation
- > 50.1 GB: Rejected by file size validation
Part calculation logic:
- If file <= 250MB target part: single part upload
- Otherwise: divide into up to 250 parts at 250MB each
- If calculated part size < 5MB: use 5MB minimum and recalculate part count
Concurrency pattern:
- Sliding window with 4 concurrent uploads
- New parts submitted only when previous parts complete (backpressure)
- Parts read sequentially from disk, results sorted by part number
Reasoning
S3 requires a minimum part size of 5MB and a maximum of 10,000 parts per upload. The SDK conservatively limits to 250 parts maximum. The 250MB target part size balances between too many small requests (overhead) and too few large parts (risk of timeout on slow connections). The sliding window pattern with max 4 concurrent parts prevents memory exhaustion from loading too much of a large file simultaneously.
Constants from `src/together/constants.py:20-30`:
MAX_CONCURRENT_PARTS = 4
MIN_PART_SIZE_MB = 5 # S3 requirement
TARGET_PART_SIZE_MB = 250 # Optimal part size
MAX_MULTIPART_PARTS = 250 # Maximum parts per upload
MULTIPART_UPLOAD_TIMEOUT = 300 # 5 minutes per part
MULTIPART_THRESHOLD_GB = 5.0 # Threshold for multipart
MAX_FILE_SIZE_GB = 50.1 # Absolute max file size
Download uses 10MB blocks from `src/together/constants.py:16`:
DOWNLOAD_BLOCK_SIZE = 10 * 1024 * 1024 # 10 MB streaming blocks