Implementation:Huggingface Datatrove MergeStatsTool
| Knowledge Sources | |
|---|---|
| Domains | Pipeline Operations, Statistics |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
MergeStatsTool is a command-line utility that combines per-task statistics files from a distributed pipeline run into a single merged statistics file.
Description
The merge_stats module provides a simple but essential post-processing tool for distributed pipeline runs. When a pipeline is executed across multiple tasks, each task produces its own statistics file (as JSON). This tool reads all statistics files from a specified folder, deserializes each into a PipelineStats object, and combines them using the PipelineStats addition operator. The merged result is then saved to an output file (defaulting to `merged_stats.json`).
The tool uses tqdm for progress bars during both the file loading and merging phases, providing visibility into the process for large numbers of statistics files. The merging starts with an empty PipelineStats instance and accumulates all individual stats through summation. The final merged statistics are both written to disk and logged for immediate inspection.
Usage
Use this tool after a distributed pipeline run completes to aggregate per-task statistics into a single comprehensive summary. This is useful for generating final reports, comparing pipeline runs, or feeding combined statistics into downstream analysis.
Code Reference
Source Location
- Repository: Huggingface_Datatrove
- File: src/datatrove/tools/merge_stats.py
- Lines: 1-44
Signature
def main():
"""Combine and average per task statistics into a single file."""
Import
from datatrove.tools.merge_stats import main
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| path | str (CLI argument) | No | Path to the folder containing per-task statistics JSON files (default: current directory) |
| --output / -o | str | No | Output file path for the merged statistics (default: "merged_stats.json") |
Outputs
| Name | Type | Description |
|---|---|---|
| merged_stats.json | JSON file | Single JSON file containing the combined statistics from all tasks |
| Log output | Logger | Merged statistics summary printed to the logger |
Usage Examples
Basic Usage
# Merge stats from a specific folder
python -m datatrove.tools.merge_stats /path/to/stats/ -o merged_stats.json
# Programmatic usage
from datatrove.tools.merge_stats import main
import sys
sys.argv = ["merge_stats", "/path/to/stats/", "-o", "output_stats.json"]
main()