Implementation:Huggingface Datatrove InferenceDatasetCardGenerator

Knowledge Sources	InferenceDatasetCardGenerator source InferenceDatasetCardParams source Card builder functions
Domains	Documentation, Data_Governance, HuggingFace Hub
Last Updated	2026-02-14 00:00 GMT

Overview

Pipeline step that generates and uploads HuggingFace dataset cards with model provenance, generation parameters, and processing statistics for synthetic datasets.

Description

Template:Code is a Template:Code dataclass that wraps the card generation and upload logic. It takes an Template:Code object containing all metadata needed to produce a complete dataset card, then:

Passes through any input data (acts as a transparent pipeline step)
On rank 0 only, calls Template:Code to generate and upload the final card
Handles errors gracefully with warning-level logging (card upload failure does not fail the pipeline)

The Template:Code dataclass captures:

output_repo_id: Target HuggingFace Hub repository for the card upload
input_dataset_name/split/config: Source dataset identification
prompt_column/prompt_template/system_prompt: Prompt configuration details
model_name/model_revision: Model identification for provenance
generation_kwargs: Generation parameters (temperature, top_p, max_tokens, etc.)
spec_config: Optional speculative decoding configuration
stats_path: Path to the Template:Code file for loading processing statistics

The Template:Code function handles the full workflow:

Load job statistics from Template:Code (with up to 5-minute polling wait)
Fetch source dataset metadata from HuggingFace Hub API (license, languages, tags)
Render the card from a template with placeholder substitution
Upload to the Hub via Template:Code

Supporting classes and functions include:

JobStats: Dataclass holding document count, mean doc length, and token statistics
load_job_stats(): Loads and parses statistics from the executor's JSON output
fetch_source_dataset_metadata(): Queries HuggingFace Hub for source dataset card data
format_number(): Human-readable number formatting with K/M/B/T suffixes

Usage

Use InferenceDatasetCardGenerator when:

Synthetic data generation is complete and a final dataset card is needed
The generated data is being uploaded to HuggingFace Hub and needs proper documentation
Provenance tracking for model, parameters, and source data is required

Code Reference

Source Location

Repository: huggingface/datatrove
File: src/datatrove/pipeline/inference/dataset_card_generator.py:L34-63

Signature

@dataclass
class InferenceDatasetCardParams:
    output_repo_id: str
    input_dataset_name: str
    input_dataset_split: str
    input_dataset_config: str | None
    prompt_column: str
    prompt_template: str | None
    system_prompt: str | None
    model_name: str
    model_revision: str
    generation_kwargs: dict[str, Any]
    spec_config: str | None
    stats_path: str


@dataclass
class InferenceDatasetCardGenerator(PipelineStep):
    params: InferenceDatasetCardParams

    name: str = "InferenceDatasetCardGenerator"
    type: str = "Generator"

    def run(self, data=None, rank: int = 0, world_size: int = 1):
        """Generate final dataset card after all data is processed. Only runs on rank 0."""
        ...


def build_and_upload_dataset_card(
    *,
    params: InferenceDatasetCardParams,
    progress_section: str = "",
) -> None:
    """Build and upload dataset card to HuggingFace Hub."""
    ...

Import

from datatrove.pipeline.inference.dataset_card_generator import (
    InferenceDatasetCardGenerator,
    InferenceDatasetCardParams,
)

I/O Contract

Inputs

Name	Type	Required	Description
params	InferenceDatasetCardParams	Yes	Dataclass containing all metadata needed for card generation (repo ID, model info, source dataset info, stats path)
data	Iterable / None	No	Optional passthrough data from previous pipeline step
rank	int	Yes	Process rank; card generation only executes on rank 0
world_size	int	Yes	Total number of processes (used for rank check)

Outputs

Name	Type	Description
HuggingFace dataset card	README.md file	Uploaded to Template:Code on HuggingFace Hub with YAML front matter and formatted documentation
Passthrough data	Iterable	Input data yielded unchanged (transparent pipeline step)

Usage Examples

Example: Adding card generation to an inference pipeline

from datatrove.pipeline.inference.dataset_card_generator import (
    InferenceDatasetCardGenerator,
    InferenceDatasetCardParams,
)

card_params = InferenceDatasetCardParams(
    output_repo_id="my-org/synthetic-summaries",
    input_dataset_name="my-org/source-documents",
    input_dataset_split="train",
    input_dataset_config=None,
    prompt_column="text",
    prompt_template="Summarize the following:\n{text}",
    system_prompt="You are a helpful summarization assistant.",
    model_name="meta-llama/Llama-3.1-70B-Instruct",
    model_revision="main",
    generation_kwargs={
        "temperature": 0.3,
        "top_p": 0.95,
        "max_tokens": 512,
        "model_max_context": 8192,
    },
    spec_config=None,
    stats_path="/path/to/stats.json",
)

# Use as pipeline step
card_generator = InferenceDatasetCardGenerator(params=card_params)

Example: Directly building and uploading a card

from datatrove.pipeline.inference.dataset_card_generator import (
    InferenceDatasetCardParams,
    build_and_upload_dataset_card,
)

params = InferenceDatasetCardParams(
    output_repo_id="my-org/synthetic-dataset",
    input_dataset_name="my-org/raw-data",
    input_dataset_split="train",
    input_dataset_config=None,
    prompt_column="content",
    prompt_template=None,
    system_prompt=None,
    model_name="meta-llama/Llama-3.1-8B-Instruct",
    model_revision="main",
    generation_kwargs={"temperature": 0.7, "max_tokens": 1024},
    spec_config=None,
    stats_path="/output/stats.json",
)

# Generate and upload the final card
build_and_upload_dataset_card(params=params, progress_section="")

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment