Implementation:Microsoft DeepSpeedExamples BingBert GlueClassifierLarge

Knowledge Sources	Microsoft_DeepSpeedExamples
Domains	BERT Finetuning, NLU Benchmarks
Last Updated	2026-02-07 12:00 GMT

Overview

GLUE benchmark finetuning runner for BERT-large models with DeepSpeed, including checkpointing utilities for resumable distributed training.

Description

This module implements the GLUE benchmark finetuning pipeline for BERT-large models using DeepSpeed. It extends the base GLUE classifier with additional checkpoint management utilities (checkpoint_model and load_checkpoint) to support resumable training, which is essential for the longer training times required by BERT-large models.

The script includes the same comprehensive set of GLUE task processors as the BERT-base variant: MRPC, MNLI (matched and mismatched), CoLA, SST-2, STS-B, QQP, QNLI, RTE, and WNLI. Each processor inherits from the DataProcessor base class and implements task-specific data loading, label definition, and example creation methods.

The checkpoint utilities use the DeepSpeed model's save_checkpoint and load_checkpoint methods to persist and restore the full training state including model weights, optimizer state, epoch number, global step count, and data sample counts. This enables fault-tolerant training of large models across multiple GPUs with automatic recovery from interruptions.

Usage

Use this script to finetune BERT-large models on GLUE benchmark tasks with DeepSpeed. It is particularly suitable for large-scale distributed training where checkpointing and recovery are essential due to longer training times.

Code Reference

Source Location

• Repository: Microsoft_DeepSpeedExamples
• File: training/bing_bert/run_glue_classifier_bert_large.py
• Lines: 1-1282

Signature

def checkpoint_model(PATH, ckpt_id, model, epoch, last_global_step, last_global_data_samples, **kwargs)
def load_checkpoint(model, PATH, ckpt_id)

class InputExample(object)
class InputFeatures(object)
class DataProcessor(object)
class MrpcProcessor(DataProcessor)
class MnliProcessor(DataProcessor)
class MnliMismatchedProcessor(MnliProcessor)
class ColaProcessor(DataProcessor)
class Sst2Processor(DataProcessor)
class StsbProcessor(DataProcessor)
class QqpProcessor(DataProcessor)
class QnliProcessor(DataProcessor)
class RteProcessor(DataProcessor)
class WnliProcessor(DataProcessor)

Import

from run_glue_classifier_bert_large import (
    InputExample, InputFeatures, DataProcessor,
    MrpcProcessor, MnliProcessor, checkpoint_model,
    load_checkpoint
)

I/O Contract

Inputs

Name	Type	Required	Description
--task_name	str	Yes	Name of the GLUE task (MRPC, MNLI, CoLA, SST-2, STS-B, QQP, QNLI, RTE, WNLI)
--data_dir	str	Yes	Directory containing GLUE task data files
--bert_model	str	Yes	Pretrained BERT-large model name or path (e.g., bert-large-uncased)
--output_dir	str	Yes	Directory for saving model checkpoints and predictions
--max_seq_length	int	No	Maximum input sequence length after tokenization, default 128
--train_batch_size	int	No	Total training batch size across all GPUs
--learning_rate	float	No	Initial learning rate for BertAdam optimizer
--num_train_epochs	float	No	Total number of training epochs
PATH	str	Yes	Checkpoint save/load path for checkpoint_model/load_checkpoint
ckpt_id	str	Yes	Checkpoint identifier for versioning saved states

Outputs

Name	Type	Description
eval_accuracy	float	Classification accuracy on the dev set
eval_loss	float	Evaluation loss on the dev set
eval_f1	float	F1 score for MRPC and QQP tasks
eval_mcc	float	Matthews correlation coefficient for CoLA task
eval_pearson	float	Pearson correlation for STS-B regression task
checkpoint	files	Saved model weights, optimizer state, and training metadata

Usage Examples

# Command-line usage with DeepSpeed for BERT-large
# deepspeed run_glue_classifier_bert_large.py \
#     --task_name MNLI \
#     --data_dir /data/glue/MNLI \
#     --bert_model bert-large-uncased \
#     --output_dir /output/mnli_large \
#     --max_seq_length 128 \
#     --train_batch_size 32 \
#     --learning_rate 1e-5 \
#     --num_train_epochs 3.0 \
#     --deepspeed --deepspeed_config ds_config_large.json

# Checkpoint management
checkpoint_model(
    PATH="/output/mnli_large/checkpoints",
    ckpt_id="step_5000",
    model=deepspeed_model,
    epoch=1,
    last_global_step=5000,
    last_global_data_samples=160000
)