Implementation:Huggingface Datasets Pdf

Overview

Pdf is an experimental feature type for handling PDF documents in datasets. Implemented as a dataclass, it provides encoding, decoding, storage casting, and embedding capabilities for PDF files using pdfplumber integration. PDFs can be loaded from file paths, raw bytes, dictionary representations, or pdfplumber.pdf.PDF objects. The underlying Arrow storage format uses a struct with bytes (binary) and path (string) fields.

Source File

Property	Value
Repository	huggingface/datasets
File	src/datasets/features/pdf.py
Lines	280
Domain	Document_Processing, Data_Processing

Import

from datasets import Pdf
# or
from datasets.features import Pdf

Class: Pdf

Type: @dataclass

Constructor

@dataclass
class Pdf:
    decode: bool = True
    id: Optional[str] = field(default=None, repr=False)

Parameter	Type	Default	Description
decode	bool	True	Whether to decode PDF data into pdfplumber.pdf.PDF objects. If False, returns raw dictionaries with path and bytes keys.
id	Optional[str]	None	Optional identifier for the feature (not shown in repr).

Class Variables

Variable	Value	Description
dtype	"pdfplumber.pdf.PDF"	The string representation of the decoded type
pa_type	pa.struct({"bytes": pa.binary(), "path": pa.string()})	The PyArrow storage type
_type	"Pdf"	Internal type identifier (not configurable)

Methods

__call__()

Returns the PyArrow storage type (pa_type).

encode_example(value)

Encodes a PDF input into the Arrow-compatible dictionary format. Accepts the following input types:

Input Type	Behavior
str	Treated as a file path; returns {"path": value, "bytes": None}
pathlib.Path	Converted to absolute path string; returns {"path": str(value.absolute()), "bytes": None}
bytes / bytearray	Stored as raw bytes; returns {"path": None, "bytes": value}
pdfplumber.pdf.PDF	Encoded via encode_pdfplumber_pdf()
dict with path (local file)	Returns with bytes set to None to avoid duplication
dict with bytes or path	Passes through the provided values

def encode_example(self, value: Union[str, bytes, bytearray, dict, "pdfplumber.pdf.PDF"]) -> dict:
    if config.PDFPLUMBER_AVAILABLE:
        import pdfplumber
    else:
        pdfplumber = None

    if isinstance(value, str):
        return {"path": value, "bytes": None}
    elif isinstance(value, Path):
        return {"path": str(value.absolute()), "bytes": None}
    elif isinstance(value, (bytes, bytearray)):
        return {"path": None, "bytes": value}
    elif pdfplumber is not None and isinstance(value, pdfplumber.pdf.PDF):
        return encode_pdfplumber_pdf(value)
    elif value.get("path") is not None and os.path.isfile(value["path"]):
        return {"bytes": None, "path": value.get("path")}
    elif value.get("bytes") is not None or value.get("path") is not None:
        return {"bytes": value.get("bytes"), "path": value.get("path")}
    else:
        raise ValueError(
            f"A pdf sample should have one of 'path' or 'bytes' but they are missing or None in {value}."
        )

decode_example(value, token_per_repo_id=None)

Decodes a stored PDF entry back into a pdfplumber.pdf.PDF object. Handles local files, remote Hub files (with token-based authentication), and in-memory bytes.

• Raises RuntimeError if decode is False.
• Raises ImportError if pdfplumber is not installed.
• For remote files, resolves Hub URLs and applies per-repository authentication tokens.

flatten()

Returns the feature itself if decode is True. Otherwise, returns a dictionary with "bytes" mapped to Value("binary") and "path" mapped to Value("string").

def flatten(self) -> Union["FeatureType", Dict[str, "FeatureType"]]:
    from .features import Value
    return (
        self
        if self.decode
        else {
            "bytes": Value("binary"),
            "path": Value("string"),
        }
    )

cast_storage(storage)

Casts an Arrow array to the Pdf storage type. Supports conversion from the following Arrow types:

Arrow Type	Conversion
pa.string()	Treated as path data; bytes set to None
pa.binary()	Treated as byte data; path set to None
pa.struct with bytes and/or path fields	Fields extracted and restructured

embed_storage(storage, token_per_repo_id=None)

Embeds PDF files into the Arrow array by reading remote file contents into bytes. For each entry, if bytes is None, the file at path is downloaded and read. Paths are reduced to basenames after embedding.

Helper Functions

pdf_to_bytes(pdf)

Converts a pdfplumber.pdf.PDF object to bytes by writing each page's stream to a buffer.

def pdf_to_bytes(pdf: "pdfplumber.pdf.PDF") -> bytes:
    with BytesIO() as buffer:
        for page in pdf.pages:
            buffer.write(page.pdf.stream)
        return buffer.getvalue()

encode_pdfplumber_pdf(pdf)

Encodes a pdfplumber.pdf.PDF into a dictionary. If the PDF has an associated file path (via pdf.stream.name), returns the path. Otherwise, serializes the PDF content to bytes.

def encode_pdfplumber_pdf(pdf: "pdfplumber.pdf.PDF") -> dict:
    if hasattr(pdf, "stream") and hasattr(pdf.stream, "name") and pdf.stream.name:
        return {"path": pdf.stream.name, "bytes": None}
    else:
        return {"path": None, "bytes": pdf_to_bytes(pdf)}

I/O

Direction	Description
Input	PDF file paths (str, Path), raw bytes, pdfplumber.pdf.PDF objects, or dictionaries with path/bytes keys
Output	When decoding: pdfplumber.pdf.PDF objects. When not decoding: dictionaries with path and bytes fields. Arrow storage: pa.struct({"bytes": pa.binary(), "path": pa.string()})

Dependencies

Module	Purpose
pyarrow	Arrow array types and struct definitions
pdfplumber	PDF parsing and reading (optional, required for decoding)
datasets.config	Hub endpoint and feature availability configuration
datasets.download.download_config.DownloadConfig	Token-based download configuration
datasets.table.array_cast	Arrow array type casting
datasets.utils.file_utils	Local path detection and file opening
datasets.utils.py_utils	Null value handling and URL pattern matching

Usage

from datasets import Dataset, Pdf

# Create a dataset with PDF files
ds = Dataset.from_dict({"pdf": ["path/to/file.pdf"]}).cast_column("pdf", Pdf())

# Access a decoded PDF (returns pdfplumber.pdf.PDF)
pdf_obj = ds[0]["pdf"]

# Use without decoding (returns dict with path/bytes)
ds_raw = ds.cast_column("pdf", Pdf(decode=False))
print(ds_raw[0]["pdf"])
# {'bytes': None, 'path': 'path/to/file.pdf'}

Related Pages

• Huggingface_Datasets_Translation