Implementation:Microsoft DeepSpeedExamples GptFinetuning AnalyzeData

Knowledge Sources	Microsoft_DeepSpeedExamples
Domains	Data Analysis, Data Efficiency
Last Updated	2026-02-07 12:00 GMT

Overview

Data analysis script for GPT finetuning that tokenizes and analyzes training datasets using DeepSpeed's DataAnalyzer for data-efficient training.

Description

This module implements a data analysis pipeline for GPT finetuning that leverages DeepSpeed's DataAnalyzer to examine training data characteristics before the actual training process. It loads datasets from HuggingFace's datasets library or local files, tokenizes them using AutoTokenizer, and groups the tokenized texts into fixed-length blocks suitable for causal language modeling.

The script uses DeepSpeed's data efficiency components including DataAnalyzer for statistical analysis of the training data and MMapIndexedDataset for memory-mapped dataset access. This analysis step is a prerequisite for data-efficient training strategies such as curriculum learning and data sampling, which can reduce training time and improve model quality.

The main function handles the complete pipeline: dataset loading (from hub or local files), configuration setup (including support for untied word and relative position embeddings), tokenization with configurable preprocessing workers, text grouping into fixed block sizes, and invocation of the DataAnalyzer on the processed dataset.

Usage

Use this script as a preprocessing step before GPT finetuning when employing DeepSpeed's data efficiency features. Run it to analyze the training data and generate indices required by curriculum learning or other data sampling strategies.

Code Reference

Source Location

• Repository: Microsoft_DeepSpeedExamples
• File: training/data_efficiency/gpt_finetuning/analyze_data.py
• Lines: 1-404

Signature

def parse_args()
def main()

Import

from analyze_data import parse_args, main

I/O Contract

Inputs

Name	Type	Required	Description
--model_name_or_path	str	Yes	Path to pretrained model or HuggingFace model identifier for tokenizer and config
--dataset_name	str	No	Name of a HuggingFace dataset to load (mutually exclusive with --train_file)
--dataset_config_name	str	No	Configuration name for the HuggingFace dataset
--train_file	str	No	Path to a local CSV, JSON, or text file containing training data
--validation_file	str	No	Path to a local file containing validation data
--validation_split_percentage	int	No	Percentage of training data to use as validation if no split exists, default 5
--block_size	int	No	Input sequence length after tokenization, defaults to model max length
--preprocessing_num_workers	int	No	Number of processes for parallel data preprocessing
--per_device_train_batch_size	int	No	Batch size per device for training, default 8
--output_dir	str	No	Directory to store the final model and analysis results

Outputs

Name	Type	Description
tokenized_datasets	Dataset	HuggingFace dataset with tokenized and grouped text ready for analysis
analysis_results	files	Data analysis output from DeepSpeed DataAnalyzer stored in output_dir
indexed_dataset	MMapIndexedDataset	Memory-mapped indexed dataset for efficient data access during training

Usage Examples

# Command-line usage for analyzing a HuggingFace dataset
# python analyze_data.py \
#     --model_name_or_path gpt2 \
#     --dataset_name wikitext \
#     --dataset_config_name wikitext-2-raw-v1 \
#     --block_size 1024 \
#     --output_dir /output/analysis \
#     --per_device_train_batch_size 8

# Command-line usage for analyzing a local text file
# python analyze_data.py \
#     --model_name_or_path gpt2 \
#     --train_file /data/train.txt \
#     --validation_file /data/valid.txt \
#     --block_size 1024 \
#     --output_dir /output/analysis