Principle:Huggingface Datatrove Pipeline Failure Diagnosis
| Knowledge Sources | |
|---|---|
| Domains | Pipeline Operations, Observability |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
Pipeline failure diagnosis is the practice of systematically identifying failed tasks in a distributed pipeline run and retrieving their logs to determine root causes of failure.
Description
In distributed data processing, pipelines are divided into many independent tasks running across multiple workers. When some tasks fail, operators need to quickly identify which tasks did not complete and access their logs to understand the failure. Pipeline failure diagnosis provides a structured approach to this problem by leveraging the pipeline's own metadata (task counts from executor configuration) and completion tracking (per-task completion markers) to compute the set of failed tasks, then correlating those with available log files.
This principle emphasizes automated failure detection over manual inspection. Rather than requiring operators to manually check each task's status or search through hundreds of log files, the diagnostic tooling cross-references expected tasks against observed completions and surfaces only the relevant information.
Usage
Apply this principle after any distributed pipeline run that has incomplete tasks. It is especially valuable in long-running jobs with hundreds or thousands of tasks, where manual investigation would be prohibitively time-consuming.
Theoretical Basis
Pipeline failure diagnosis relies on several operational concepts:
Completion tracking: Each task, upon successful completion, writes a marker file to a completions directory (named by its rank, e.g., `completions/00042`). The set difference between all expected ranks (0 to world_size-1) and the set of observed completion markers yields the set of incomplete tasks. This is a lightweight, filesystem-based protocol that works across all storage backends.
Log correlation: Task log files follow a naming convention that encodes the task rank (e.g., `logs/task_00042.log`). By parsing the rank from each log filename using a regex pattern and filtering against the set of incomplete ranks, the diagnostic tool retrieves only the logs that correspond to failed tasks, avoiding information overload.
Interactive triage: When multiple tasks have failed, the logs are presented one at a time with interactive prompts allowing the operator to stop early once the root cause is identified. This is more efficient than dumping all failed logs at once, as many failures often share a common cause.
Configuration-driven expectations: The total number of expected tasks is read from the executor.json configuration file, ensuring that the diagnostic tool has an authoritative source of truth for what constitutes a complete run, independent of any external job scheduler or orchestration system.