Implementation:Huggingface Datatrove ParagraphStats
| Knowledge Sources | |
|---|---|
| Domains | Data Quality, Statistics |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
ParagraphStats is a statistics pipeline step that computes paragraph-level structural metrics for documents, including counts, length distributions, and duplication ratios.
Description
ParagraphStats extends BaseStats to analyze the paragraph structure of documents. Paragraphs are identified by splitting document text on double newlines (\n\n). The class computes several metrics: the total number of paragraphs (n_paragraphs), the average paragraph length in characters (avg_paragraph_length), the ratio of paragraphs below configurable character thresholds (short_paragraph_ratio_{chars}), and the ratio of paragraphs above configurable character thresholds (long_paragraph_ratio_{chars}).
Additionally, the class leverages the find_duplicates function from the Gopher repetition filter to detect duplicate paragraphs. It reports both paragraph_duplicates (the fraction of paragraphs that are exact duplicates of other paragraphs) and paragraph_char_duplicates (the fraction of total characters contained in duplicate paragraphs). These duplication metrics are valuable quality signals, as high paragraph duplication often indicates boilerplate content, template-generated text, or extraction artifacts.
The ignore_empty_paragraphs option controls whether empty paragraphs (containing only whitespace) are included in the length-based calculations. The count metric always includes all paragraphs regardless of this setting.
Usage
Use ParagraphStats when you need to profile the structural quality of documents at the paragraph level. It is particularly useful for detecting boilerplate-heavy documents, identifying excessively fragmented or monolithic content, and assessing paragraph-level duplication in web-crawled datasets.
Code Reference
Source Location
- Repository: Huggingface_Datatrove
- File: src/datatrove/pipeline/stats/paragraph_stats.py
- Lines: 1-74
Signature
class ParagraphStats(BaseStats):
type = "📊 - STATS"
name = "📄 Paragraph stats"
def __init__(
self,
output_folder: DataFolderLike,
short_paragraph_max_chars_threshold: list[int] | None = None,
long_paragraph_max_chars_threshold: list[int] | None = None,
ignore_empty_paragraphs: bool = False,
histogram_round_digits: int = 3,
groups_to_compute: list[GROUP] = list(get_args(GROUP)),
top_k_config: TopKConfig = DEFAULT_TOP_K_CONFIG,
) -> None
def extract_stats(self, doc: Document) -> dict[str, int | float]:
...
Import
from datatrove.pipeline.stats.paragraph_stats import ParagraphStats
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| output_folder | DataFolderLike | Yes | Folder where computed statistics will be saved |
| short_paragraph_max_chars_threshold | list[int] or None | No | Character thresholds for classifying short paragraphs (default: [100]) |
| long_paragraph_max_chars_threshold | list[int] or None | No | Character thresholds for classifying long paragraphs (default: [1000]) |
| ignore_empty_paragraphs | bool | No | Whether to exclude empty paragraphs from length calculations (default: False) |
| histogram_round_digits | int | No | Decimal digits for histogram rounding (default: 3) |
| groups_to_compute | list[GROUP] | No | Grouping strategies for statistics (default: all groups) |
| top_k_config | TopKConfig | No | Top-K configuration for high-cardinality groups |
Outputs
| Name | Type | Description |
|---|---|---|
| n_paragraphs | int | Total number of non-empty paragraphs in the document |
| avg_paragraph_length | float | Average character length of paragraphs |
| short_paragraph_ratio_{chars} | float | Fraction of paragraphs with length at or below the threshold |
| long_paragraph_ratio_{chars} | float | Fraction of paragraphs with length at or above the threshold |
| paragraph_duplicates | float | Fraction of paragraphs that are duplicates of other paragraphs |
| paragraph_char_duplicates | float | Fraction of total characters in duplicate paragraphs |
Usage Examples
Basic Usage
from datatrove.pipeline.stats.paragraph_stats import ParagraphStats
stats = ParagraphStats(
output_folder="output/stats/",
)
Custom Thresholds
from datatrove.pipeline.stats.paragraph_stats import ParagraphStats
stats = ParagraphStats(
output_folder="output/stats/",
short_paragraph_max_chars_threshold=[50, 100, 200],
long_paragraph_max_chars_threshold=[500, 1000, 2000],
ignore_empty_paragraphs=True,
)