Heuristic:Wandb Weave Concurrency Deadlock Prevention
| Knowledge Sources | |
|---|---|
| Domains | Concurrency, Debugging, Performance |
| Last Updated | 2026-02-14 12:00 GMT |
Overview
Thread pool architecture with deadlock-safe task submission, dual executor pools, and context-aware threading for trace operations.
Description
Weave uses a dual-executor architecture for background operations: a main executor for general deferred tasks (call processing, serialization) and a fast-lane executor for operations guaranteed not to spawn child tasks (file uploads). This separation, combined with the `_make_deadlock_safe()` wrapper pattern, prevents deadlocks that would occur when worker threads submit new tasks to their own pool and then wait for results. Additional safety mechanisms include per-instance PIL Image locks for thread-safe image loading, double-checked locking for client initialization, and `ContextVar`-based thread context propagation.
Usage
Use this heuristic when debugging hangs or deadlocks in Weave tracing, when configuring parallelism (`WEAVE_PARALLELISM` or `client_parallelism`), or when writing custom ops that perform I/O in background threads. Critical for understanding why setting `client_parallelism=0` forces synchronous execution.
The Insight (Rule of Thumb)
- Action: Never wait on futures from within a thread pool worker context. Use the `_make_deadlock_safe()` wrapper for all submitted tasks.
- Value: The wrapper ensures child operations from worker threads don't block the parent pool.
- Trade-off: Slightly more complex code but eliminates an entire class of deadlock bugs.
- Action: Use the dual-executor split: `BACKGROUND_PARALLELISM_MIX = 0.5` allocates half the workers to the main pool and half to the fast-lane pool.
- Value: Main pool handles general tasks; fast-lane handles file uploads (no child tasks possible).
- Trade-off: Splitting reduces per-pool parallelism but eliminates cross-pool deadlocks.
- Action: Set `client_parallelism=0` to force synchronous execution for debugging.
- Value: All background operations execute immediately in the main thread.
- Trade-off: Significant performance degradation but eliminates all concurrency-related issues for debugging.
- Action: Default parallelism is 20 workers total (via `WEAVE_PARALLELISM`), split between main and fast-lane.
- Value: Configurable via environment variable or `UserSettings.client_parallelism`.
- Trade-off: More workers handle higher throughput but consume more system resources and threads.
Reasoning
The deadlock scenario occurs when: (1) a worker thread submits a new task to its own pool, (2) all workers are busy, (3) the new task waits in the queue, (4) the worker that submitted it is also waiting for the result, creating a circular dependency. The `_make_deadlock_safe()` pattern breaks this cycle by ensuring that callbacks from futures are executed outside the thread pool context.
The `ContextAwareThreadPoolExecutor` propagates Python `ContextVar` values to worker threads, ensuring that tracing context (call stack, retry IDs, settings) is maintained across thread boundaries.
Code Evidence
Dual executor architecture from `weave/trace/weave_client.py:288-303`:
BACKGROUND_PARALLELISM_MIX = 0.5
class WeaveClient:
# Main future executor, handling deferred tasks for the client
future_executor: FutureExecutor
# Fast-lane executor for operations guaranteed to not defer
# to child operations, impossible to deadlock
# Currently only used for create_file operation
future_executor_fastlane: FutureExecutor | None
FutureExecutor zero-workers synchronous mode from `weave/trace/concurrent/futures.py:67-80`:
class FutureExecutor:
def __init__(
self,
max_workers: int | None = None,
thread_name_prefix: str = THREAD_NAME_PREFIX,
):
self._max_workers = max_workers
self._executor: ContextAwareThreadPoolExecutor | None = None
if max_workers != 0:
self._executor = ContextAwareThreadPoolExecutor(
max_workers=max_workers, thread_name_prefix=thread_name_prefix
)
Default parallelism setting from `weave/trace/env.py:44-45`:
def get_weave_parallelism() -> int:
return int(os.getenv(WEAVE_PARALLELISM, "20"))
Parallelism setting override from `weave/trace/settings.py:113-123`:
client_parallelism: int | None = None
"""
Sets the number of workers to use for background operations.
If not set, automatically adjusts based on the number of cores.
Setting this to 0 will effectively execute all background operations
immediately in the main thread.
This cannot be changed after the client has been initialized.
"""