Implementation:Microsoft DeepSpeedExamples PretrainedBert FileUtils

Knowledge Sources	Microsoft_DeepSpeedExamples
Domains	Caching, File Management
Last Updated	2026-02-07 12:00 GMT

Overview

A file caching and download utility that provides URL-to-local-path resolution with support for HTTP, HTTPS, and S3 sources, including ETag-based cache invalidation.

Description

file_utils.py (in the pytorch_pretrained_bert package) provides a comprehensive caching system for downloading and managing pretrained model files. The central function cached_path determines whether a given string is a URL or local file path, and for URLs it transparently downloads and caches the resource. The cache directory defaults to ~/.pytorch_pretrained_bert and can be overridden via the PYTORCH_PRETRAINED_BERT_CACHE environment variable.

The caching system uses SHA-256 hashing of URLs and ETags to generate unique filenames. Each cached file is accompanied by a .json metadata sidecar containing the original URL and ETag, enabling filename_to_url reverse lookups. Download integrity is ensured by writing to a temporary file first and copying to the cache only after successful completion, preventing corrupt cache entries from interrupted downloads.

The module supports three download protocols: HTTP/HTTPS via the requests library with streaming and tqdm progress bars, and S3 via boto3 with dedicated s3_get and s3_etag functions. The s3_request decorator provides enhanced error handling that translates S3 ClientError 404 responses into Python FileNotFoundError exceptions. Additional utility functions include read_set_from_file for loading deduplicated text collections and get_file_extension for path extension extraction.

Usage

Use this module when you need to resolve pretrained BERT model URLs to local cached files. It is used internally by the modeling module to load pretrained weights and configurations from remote sources. The cached_path function is the primary entry point for obtaining local file paths from URLs.

Code Reference

Source Location

• Repository: Microsoft_DeepSpeedExamples
• File: training/bing_bert/pytorch_pretrained_bert/file_utils.py
• Lines: 1-246

Signature

PYTORCH_PRETRAINED_BERT_CACHE = Path(
    os.getenv('PYTORCH_PRETRAINED_BERT_CACHE',
              Path.home() / '.pytorch_pretrained_bert'))

def url_to_filename(url: str, etag: str = None) -> str:
def filename_to_url(filename: str, cache_dir: Union[str, Path] = None) -> Tuple[str, str]:
def cached_path(url_or_filename: Union[str, Path], cache_dir: Union[str, Path] = None) -> str:
def split_s3_path(url: str) -> Tuple[str, str]:
def s3_request(func: Callable):
def s3_etag(url: str) -> Optional[str]:
def s3_get(url: str, temp_file: IO) -> None:
def http_get(url: str, temp_file: IO) -> None:
def get_from_cache(url: str, cache_dir: Union[str, Path] = None) -> str:
def read_set_from_file(filename: str) -> Set[str]:
def get_file_extension(path: str, dot=True, lower: bool = True):

Import

from pytorch_pretrained_bert.file_utils import cached_path, PYTORCH_PRETRAINED_BERT_CACHE

I/O Contract

Inputs

Name	Type	Required	Description
url_or_filename	Union[str, Path]	Yes	A URL (http/https/s3) or local file path to resolve
cache_dir	Union[str, Path]	No	Override cache directory; defaults to PYTORCH_PRETRAINED_BERT_CACHE
url	str	Yes	URL for download functions (s3_get, http_get, s3_etag)
etag	str	No	HTTP ETag for cache key generation in url_to_filename

Outputs

Name	Type	Description
cached_path result	str	Local filesystem path to the cached file
url_to_filename result	str	SHA-256 hash-based filename for caching
filename_to_url result	Tuple[str, str]	Original URL and ETag recovered from cache metadata

Usage Examples

from pytorch_pretrained_bert.file_utils import cached_path, url_to_filename

# Resolve a URL to a local cached path (downloads if not cached)
local_path = cached_path("https://example.com/bert-base-uncased.tar.gz")

# Generate a cache filename from a URL
filename = url_to_filename("https://example.com/model.bin", etag='"abc123"')

# Use with a local file path (just validates existence)
local_path = cached_path("/path/to/local/model.bin")