Environment:FMInference FlexLLMGen HuggingFace Access

Knowledge Sources	FlexLLMGen HuggingFace Hub
Domains	Infrastructure, Model_Management
Last Updated	2026-02-09 12:00 GMT

Overview

HuggingFace Hub access environment for downloading pretrained OPT and Galactica model weights via snapshot_download, with automatic PyTorch-to-NumPy conversion.

Description

FlexLLMGen downloads pretrained model weights from the HuggingFace Hub using the huggingface_hub library's snapshot_download function. The downloaded PyTorch .bin checkpoint files are then converted to individual NumPy .npy files (one per tensor) which is the native weight format consumed by the FlexLLMGen inference engine. Supported model families include OPT (facebook/opt-*) and Galactica (facebook/galactica-*). The download process can take a long time for large models (e.g., OPT-175B has ~350GB of weights).

Usage

Use this environment when downloading model weights for the first time. FlexLLMGen automatically downloads weights from HuggingFace if no cached weights are found at the specified --path directory. After initial download and conversion, this environment is no longer needed for subsequent inference runs.

System Requirements

Category	Requirement	Notes
Network	Internet access to huggingface.co	For initial model weight download
Disk Space	Model-dependent	OPT-6.7B: ~13GB; OPT-30B: ~60GB; OPT-175B: ~350GB
CPU Memory	Model-dependent	Conversion loads PyTorch checkpoints into CPU memory

Dependencies

Python Packages

• huggingface_hub (for snapshot_download)
• torch >= 1.12 (for loading .bin checkpoints)
• numpy (for saving .npy weight files)
• tqdm (for progress bars during conversion)

Credentials

The following credentials may be required depending on model access:

• HF_TOKEN: HuggingFace API token. Required for gated models (e.g., OPT-175B requires access approval). Public models like OPT-6.7B and OPT-30B do not require authentication.

Quick Install

# Install huggingface_hub for model downloading
pip install huggingface_hub

# Optional: Login for gated model access
huggingface-cli login

# Download and convert weights (automatic when running FlexLLMGen)
python -m flexllmgen.opt_config --model opt-6.7b --path ~/opt_weights

Code Evidence

Model download via snapshot_download in flexllmgen/opt_config.py:219-233:

def download_opt_weights(model_name, path):
    from huggingface_hub import snapshot_download
    import torch

    print(f"Load the pre-trained pytorch weights of {model_name} from huggingface. "
          f"The downloading and cpu loading can take dozens of minutes. "
          f"If it seems to get stuck, you can monitor the progress by "
          f"checking the memory usage of this process.")

    if "opt" in model_name:
        hf_model_name = "facebook/" + model_name
    elif "galactica" in model_name:
        hf_model_name = "facebook/" + model_name

    folder = snapshot_download(hf_model_name, allow_patterns="*.bin")

PyTorch-to-NumPy weight conversion in flexllmgen/opt_config.py:242-254:

for bin_file in tqdm(bin_files, desc="Convert format"):
    state = torch.load(bin_file)
    for name, param in tqdm(state.items(), leave=False):
        name = name.replace("model.", "")
        name = name.replace("decoder.final_layer_norm", "decoder.layer_norm")
        param_path = os.path.join(path, name)
        with open(param_path, "wb") as f:
            np.save(f, param.cpu().detach().numpy())

Default weight path from flexllmgen/flex_opt.py:1278-1280:

parser.add_argument("--path", type=str, default="~/opt_weights",
    help="The path to the model weights. If there are no cached weights, "
         "FlexLLMGen will automatically download them from HuggingFace.")

Common Errors

Error Message	Cause	Solution
HTTPError: 401	Gated model requires authentication	Set HF_TOKEN environment variable or run huggingface-cli login
OSError: [Errno 28] No space left on device	Insufficient disk space for model weights	Free disk space; OPT-175B needs ~350GB
Download appears stuck	Large model download is slow	Monitor memory usage as suggested by the print message; check network speed

Compatibility Notes

• Supported model families: OPT (facebook/opt-*) and Galactica (facebook/galactica-*) models from HuggingFace Hub.
• Weight format: Downloaded PyTorch .bin files are converted to individual NumPy .npy files stored in a {model_name}-np subdirectory.
• Shared embeddings: The decoder.embed_tokens.weight tensor is automatically copied to lm_head.weight during conversion (tied embeddings).
• Dummy weights: For benchmarking, the system supports DUMMY_WEIGHT mode that skips downloading entirely.

Related Pages

• Implementation:FMInference_FlexLLMGen_Download_Opt_Weights