Workflow:Huggingface Datasets Format Conversion
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Interoperability, Machine_Learning |
| Last Updated | 2026-02-14 18:00 GMT |
Overview
End-to-end process for converting datasets between file formats (CSV, JSON, Parquet, SQL, text) and ML framework representations (PyTorch, TensorFlow, JAX, NumPy, Pandas, Polars).
Description
This workflow covers the bidirectional conversion capabilities of the Datasets library: importing data from various file formats and databases into the unified Arrow-backed Dataset representation, and exporting data to different file formats or ML framework tensor types. The library provides dedicated Reader and Writer classes for each file format, a pluggable Formatter system for framework-specific output conversion, and I/O methods on the Dataset class for saving and loading in different formats. The Arrow backbone enables zero-copy conversions where possible, making format changes efficient even for large datasets.
Usage
Execute this workflow when you need to ingest data from an existing source format (e.g., CSV files, JSON lines, SQL databases, Spark DataFrames), convert between storage formats (e.g., CSV to Parquet for faster reads), export datasets for use in external tools, or bridge the gap between the Dataset object and a specific ML framework's data pipeline (e.g., creating a PyTorch DataLoader or TensorFlow tf.data.Dataset).
Execution Steps
Step 1: Import from Source Format
Load data from its original format into a Dataset using format-specific readers or the universal load_dataset function with appropriate format parameters. The library supports CSV, JSON/JSONL, Parquet, Arrow IPC, plain text, SQL databases, HDF5, Lance, WebDataset archives, XML, and folder-based media layouts.
Key considerations:
- load_dataset auto-detects format from file extensions using the packaged module registry
- Each packaged module provides format-specific configuration options (CSV separators, JSON field selectors, etc.)
- Direct reader classes (CsvDatasetReader, JsonDatasetReader, etc.) offer fine-grained control
- Folder-based builders (ImageFolder, AudioFolder, VideoFolder) automatically infer labels from directory structure
- SQL and Spark readers connect directly to databases and distributed DataFrames
Step 2: Validate and Align the Schema
After importing, verify that the inferred feature types match expectations and correct any mismatches. Type inference from file formats can produce different types than intended (e.g., integer IDs loaded as strings from CSV), requiring explicit casting.
Key considerations:
- Review dataset.features to check inferred types
- Use cast() to change column types (e.g., string to int, Value to ClassLabel)
- Features can be specified upfront during loading to override inference
- Null handling varies by source format; verify null policies are correct
Step 3: Export to Target File Format
Write the dataset to a different file format using the dedicated writer methods or convenience functions on the Dataset object. Each writer handles format-specific serialization details including compression, encoding, and schema embedding.
Key considerations:
- to_csv() writes comma-separated values (configurable separator, quoting, encoding)
- to_json() writes JSON or JSON Lines format (configurable orient, compression)
- to_parquet() writes columnar Parquet with options for content-defined chunking and page indexing
- to_sql() writes to SQL databases via SQLAlchemy connections
- save_to_disk() saves the native Arrow format for fast reload with load_from_disk()
- All writers support batch_size for memory-efficient writing of large datasets
Step 4: Convert to ML Framework Tensors
Configure the Dataset to return data as framework-specific tensors for direct use in training pipelines. The Formatter system converts Arrow data to the target representation at access time, enabling seamless integration with any supported ML framework.
Supported frameworks:
- PyTorch: Returns torch.Tensor objects, compatible with DataLoader
- TensorFlow: Returns tf.Tensor objects, can create tf.data.Dataset
- JAX: Returns jax.numpy.ndarray objects
- NumPy: Returns numpy.ndarray objects
- Pandas: Returns DataFrame or Series objects
- Polars: Returns DataFrame or Series objects
- Arrow: Returns raw PyArrow tables and arrays
Step 5: Create Framework DataLoaders
Bridge the formatted Dataset into framework-specific data loading pipelines. The library provides utilities for creating PyTorch DataLoaders and TensorFlow tf.data.Datasets that wrap the formatted Dataset with batching, shuffling, and prefetching capabilities.
Key considerations:
- For PyTorch: set format to "torch" then wrap with torch.utils.data.DataLoader
- For TensorFlow: use the to_tf_dataset() utility for optimized tf.data pipeline creation
- Collation functions handle padding and batching of variable-length sequences
- DataLoader workers can access memory-mapped data efficiently
- For distributed training, combine with split_dataset_by_node for multi-GPU setups
Step 6: Combine and Interleave Datasets
Merge multiple datasets from potentially different sources into a unified dataset. Concatenation appends datasets vertically (more rows) or horizontally (more columns), while interleaving alternates between datasets with optional probability-based sampling.
Key considerations:
- concatenate_datasets combines rows (axis=0) or columns (axis=1) from multiple datasets
- interleave_datasets cycles through or samples from multiple datasets
- Both operations work with map-style and iterable datasets
- Stopping strategies control behavior when datasets have different lengths