Heuristic:Arize ai Phoenix Queue Clear Race Condition
| Knowledge Sources | |
|---|---|
| Domains | Debugging, Concurrency |
| Last Updated | 2026-02-14 06:00 GMT |
Overview
Use `.clear()` instead of list reassignment when emptying shared queues to prevent race conditions with concurrent appenders.
Description
In the Phoenix database insertion infrastructure, a queue pattern is used to batch incoming span data before writing to the database. The queue is periodically drained by an insertion loop, while producers continuously append new items. A critical tribal knowledge pattern found in the codebase warns against replacing `self._queue = []` (reassignment) with `self._queue.clear()` (in-place mutation).
Reassignment creates a new list object, meaning any concurrent code still holding a reference to the old list will append to the old (now orphaned) list, causing data loss. In-place `.clear()` modifies the same list object, so all references remain valid.
This same pattern applies to any shared mutable collection in async/concurrent Python code.
Usage
Apply this heuristic when:
- Working with shared queues or buffers in async code
- Implementing producer-consumer patterns
- Reviewing code that drains and reprocesses items from a shared list
The Insight (Rule of Thumb)
- Action: Always use `.clear()` to empty a shared list, never reassignment (`self._list = []`).
- Value: Prevents silent data loss from race conditions.
- Trade-off: None. `.clear()` is always preferable for shared mutable collections.
Reasoning
The explicit comment in `src/phoenix/db/insertion/types.py:98-100`:
async def insert(self) -> Optional[list[_DmlEventT]]:
if not self._queue:
return None
parcels = self._queue.copy()
# IMPORTANT: Use .clear() instead of reassignment, i.e. self._queue = [], to
# avoid potential race conditions when appending postponed items to the queue.
self._queue.clear()
The design also uses `.copy()` before clearing to snapshot the current items, then `.clear()` to reset the queue. This ensures that:
- The insertion loop processes a frozen snapshot of items
- New items appended during processing go to the same (now empty) list
- Postponed items that fail insertion are re-added via `self._queue.extend(items)` and will be picked up in the next drain cycle
The postpone-and-retry pattern from `types.py:108-115`:
if to_postpone:
loop = asyncio.get_running_loop()
loop.call_later(self._retry_delay_sec, self._add_postponed_to_queue, to_postpone)
This uses `call_later` to re-add failed items after a delay, which appends to the same list reference. If reassignment had been used, these items would be lost.