Implementation:Microsoft DeepSpeedExamples Bert Tokenization

Knowledge Sources	Microsoft_DeepSpeedExamples
Domains	Natural Language Processing, Tokenization
Last Updated	2026-02-07 12:00 GMT

Overview

BERT tokenization module providing end-to-end text tokenization with basic tokenization, WordPiece tokenization, and vocabulary management for BERT models.

Description

This module implements the complete BERT tokenization pipeline as originally designed by the Google AI Language Team. It provides three main tokenizer classes: BertTokenizer for end-to-end tokenization, BasicTokenizer for punctuation splitting and lowercasing, and WordpieceTokenizer for subword tokenization using a greedy longest-match-first algorithm.

The BertTokenizer class serves as the primary interface, combining basic tokenization (whitespace splitting, punctuation handling, accent stripping, CJK character isolation) with WordPiece subword tokenization. It supports loading pretrained vocabularies from local files or downloading them from the HuggingFace model hub for standard BERT variants including bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, and multilingual models.

The module also includes several utility functions for character classification (whitespace, control characters, punctuation, Chinese characters) and vocabulary loading. The tokenizer enforces maximum sequence length constraints based on the pretrained model's positional embedding size (typically 512 tokens).

Usage

Use this module when you need to tokenize text for BERT-based models in the BingBertSquad training pipeline. It is specifically designed for the custom pytorch_pretrained_bert package used in this example and handles vocabulary loading, token-to-id conversion, and id-to-token conversion for SQuAD question answering tasks.

Code Reference

Source Location

• Repository: Microsoft_DeepSpeedExamples
• File: training/BingBertSquad/pytorch_pretrained_bert/tokenization.py
• Lines: 1-386

Signature

class BertTokenizer(object):
    def __init__(self, vocab_file, do_lower_case=True, max_len=None,
                 never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")):

class BasicTokenizer(object):
    def __init__(self, do_lower_case=True,
                 never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")):

class WordpieceTokenizer(object):
    def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=100):

def load_vocab(vocab_file):
def whitespace_tokenize(text):

Import

from pytorch_pretrained_bert.tokenization import BertTokenizer, BasicTokenizer, WordpieceTokenizer

I/O Contract

Inputs

Name	Type	Required	Description
vocab_file	str	Yes	Path to a vocabulary file or a pretrained model name
do_lower_case	bool	No	Whether to lowercase the input text (default: True)
max_len	int	No	Maximum sequence length (default: None, uses model default)
never_split	tuple	No	Tokens that should never be split during tokenization
text	str	Yes	Input text string to tokenize (for tokenize method)

Outputs

Name	Type	Description
tokens	List[str]	List of wordpiece token strings from tokenize()
ids	List[int]	List of token integer IDs from convert_tokens_to_ids()
tokens_from_ids	List[str]	List of token strings from convert_ids_to_tokens()

Usage Examples

# Load a pretrained tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenize text
tokens = tokenizer.tokenize("Hello, how are you?")
# ['hello', ',', 'how', 'are', 'you', '?']

# Convert tokens to IDs
ids = tokenizer.convert_tokens_to_ids(tokens)

# Convert IDs back to tokens
recovered_tokens = tokenizer.convert_ids_to_tokens(ids)

# Load from a local vocab file
tokenizer = BertTokenizer(vocab_file='path/to/vocab.txt', do_lower_case=True)