Heuristic:Huggingface Open r1 Code Execution Timeout Strategy
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Infrastructure |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Timeout configuration strategy for code execution sandboxes using layered margins to prevent indefinite blocking while maximizing completion rates.
Description
Open-R1 uses a layered timeout strategy for code execution providers (E2B and MorphCloud). Each layer has a different timeout value with deliberate margins between them. This design ensures that: (1) the inner sandbox timeout triggers first, (2) the request-level timeout acts as a fallback, and (3) the outer asyncio timeout is the ultimate safety net. The timeout values were empirically calibrated by running 256 examples with gold solutions from the open-r1/verifiable-coding-problems-python_decontaminated dataset.
Usage
Use this heuristic when configuring code execution timeouts for GRPO training with code reward functions. Apply when setting up E2B or MorphCloud providers, or when tuning the balance between execution time and training throughput.
The Insight (Rule of Thumb)
- Action: Use a 3-layer timeout with deliberate margins between each layer.
- Value:
- E2B provider: SANDBOX_TIMEOUT = 30s, REQUEST_TIMEOUT = 28s (sandbox - 2s margin), ASYNCIO_TIMEOUT = 32s (sandbox + 2s margin).
- MorphCloud provider: SANDBOX_TIMEOUT = 90s, ASYNCIO_TIMEOUT = 96s (sandbox + 6s margin).
- MorphCloud routed: timeout = 90s, request_timeout = 96s (6s buffer larger than timeout).
- Parallelism default: 2 concurrent executions per process (matches E2B Free Hobby tier with 8 GPUs).
- Trade-off: Shorter timeouts improve training throughput but may prematurely kill correct but slow solutions. Longer timeouts improve correctness but slow down the training loop.
Reasoning
The timeout values were empirically derived:
E2B 30s timeout: Calibrated from running 256 gold-solution examples (see scripts/benchmark_e2b.py). Most correct solutions complete well within 30 seconds. The 2-second margin ensures the asyncio wrapper catches sandbox timeouts that fail to propagate properly (the comment notes "the AsyncSandbox timeout does not seem to work" reliably).
MorphCloud 90s timeout: MorphCloud instances have higher setup overhead than E2B (instance creation, file upload, compilation for C++ problems). The 3x longer timeout accounts for this overhead. The 6-second margin (vs E2B's 2-second) reflects MorphCloud's higher latency variance.
Parallelism of 2: The default of 2 parallel code executions per process was chosen for the E2B Free Hobby tier, which limits concurrent sandboxes. With 8 GPUs in a training node, this gives 16 total concurrent sandboxes.
Retry strategy (MorphCloud competitive programming): The MorphCloud client for competitive programming uses 4 retries with exponential backoff capped at 30 seconds: retry_delay = min(base_delay * (2^attempt), 30). This handles transient failures without overwhelming the service.
Code Evidence
E2B timeout calibration from src/open_r1/utils/code_providers.py:136-144:
# We set a timeout margin, as the AsyncSandbox timeout does not seem to work
# These values are based on running 256 examples with the gold solution
# from open-r1/verifiable-coding-problems-python_decontaminated
# see scripts/benchmark_e2b.py
SANDBOX_TIMEOUT = 30
MARGIN = 2
REQUEST_TIMEOUT = SANDBOX_TIMEOUT - MARGIN
ASYNCIO_TIMEOUT = SANDBOX_TIMEOUT + MARGIN
MorphCloud timeout from src/open_r1/utils/code_providers.py:284-286:
SANDBOX_TIMEOUT = 90
MARGIN = 6
ASYNCIO_TIMEOUT = SANDBOX_TIMEOUT + MARGIN
Parallelism default from src/open_r1/configs.py:286-291:
parallel_code_exec_per_proc: int = field(
default=2,
metadata={
"help": "Number of parallel E2B code executions per process. Default of 2 is suitable for the Free Hobby tier of E2B with 8 GPUs used for training."
},
)