Principle:Huggingface Optimum Hub File Management
| Knowledge Sources | |
|---|---|
| Domains | File_Management, Hub_Integration |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Pattern for transparently validating and discovering model files across both local filesystem directories and Hugging Face Hub repositories.
Description
Hub File Management addresses the problem of writing model loading and validation code that works identically whether the model is stored locally or on the Hugging Face Hub. Rather than requiring separate code paths, this principle provides unified functions that:
- Detect the source — Check if the path is a local directory or a Hub repo ID
- Local path handling — Use `os.path.isfile` and `Path.glob` for local files
- Hub path handling — Use HfApi methods (`file_exists`, `list_repo_files`) for remote repos
- Return consistent types — Always return the same type regardless of source
This enables downstream code (model exporters, loaders) to be agnostic about whether they are working with local or remote models.
Usage
Apply this principle when building model loading or validation logic that needs to support both local paths and Hub model IDs. It is used throughout Optimum for checking model file existence and discovering model artifacts.
Theoretical Basis
This follows the Adapter Pattern where a single interface adapts between two different file access mechanisms:
Pseudo-code Logic:
# Abstract algorithm (NOT real implementation)
def file_operation(model_name_or_path, ...):
if is_local_directory(model_name_or_path):
return local_filesystem_operation(model_name_or_path, ...)
else:
return hub_api_operation(model_name_or_path, ...)
The abstraction boundary is: the caller never needs to know if the model is local or remote.