Implementation:Mlc ai Mlc llm Download Cache

Overview

Download Cache is a support module in MLC LLM that provides utilities for downloading model files from HuggingFace and other URLs, with caching and integrity verification. It is located at python/mlc_llm/support/download_cache.py (237 lines).

The module implements a multi-layered caching strategy that supports read-only caches, forced re-downloads, and offline operation. It handles both Git LFS-based model repositories and direct HTTP file downloads with MD5 checksum validation.

Purpose

Large language model weights are typically hosted on HuggingFace using Git LFS. This module manages the complete lifecycle of downloading, caching, and retrieving these model files while respecting user-configurable download policies.

Download Cache Policy

The module respects the MLC_DOWNLOAD_CACHE_POLICY constant (from Constants):

Policy	Behavior
ON	Download if not cached; use cache if available
OFF	Refuse to download; raise RuntimeError
REDO	Force re-download even if cached
READONLY	Use cache only; raise RuntimeError if missing

Core Functions

git_clone

def git_clone(url: str, destination: Path, ignore_lfs: bool) -> None:

Clones a Git repository into a destination directory. The function:

1. Creates a temporary directory for the clone operation.
2. Sets GIT_LFS_SKIP_SMUDGE=1 to skip LFS file downloads during clone.
3. Clones into a .tmp subdirectory within the temp directory.
4. If ignore_lfs is False, calls git_lfs_pull to download LFS files.
5. Moves the completed clone to the final destination using shutil.move.

The use of a temporary directory ensures atomicity -- the destination only appears once the clone is fully complete.

git_lfs_pull

def git_lfs_pull(repo_dir: Path, ignore_extensions: Optional[List[str]] = None) -> None:

Downloads Git LFS-tracked files from a cloned repository. The function:

1. Lists all LFS-tracked filenames using git lfs ls-files -n.
2. Optionally filters out files matching specified extensions (e.g., .bin files).
3. Downloads each file individually with git lfs pull --include, showing progress via tqdm.

download_file

def download_file(
    url: str,
    destination: Path,
    md5sum: Optional[str],
) -> Tuple[str, Path]:

Downloads a single file from a URL with streaming (8KB chunks) and optional MD5 checksum verification:

with requests.get(url, stream=True, timeout=30) as response:
    response.raise_for_status()
    with destination.open("wb") as file:
        for chunk in response.iter_content(chunk_size=8192):
            file.write(chunk)
if md5sum is not None:
    hash_md5 = hashlib.md5()
    with destination.open("rb") as file:
        for chunk in iter(lambda: file.read(8192), b""):
            hash_md5.update(chunk)
    file_md5 = hash_md5.hexdigest()
    if file_md5 != md5sum:
        raise ValueError(...)

Returns a tuple of (url, destination) for tracking which file was downloaded.

download_and_cache_mlc_weights

def download_and_cache_mlc_weights(
    model_url: str,
    num_processes: int = 4,
    force_redo: Optional[bool] = None,
) -> Path:

The primary function for downloading and caching MLC model weights from HuggingFace. The process follows these steps:

1. Policy check: If MLC_DOWNLOAD_CACHE_POLICY is "OFF", raises an error.
2. URL parsing: Extracts user and repo from URLs starting with HF:// or https://huggingface.co/.
3. Read-only cache lookup: Checks MLC_LLM_READONLY_WEIGHT_CACHE directories for an existing mlc-chat-config.json.
4. Local cache lookup: Checks the standard cache directory at MLC_LLM_HOME/model_weights/hf/{user}/{repo}.
5. READONLY policy check: If the policy is "READONLY" and no cache is found, raises an error listing all searched paths.
6. Download: Clones the repository (skipping LFS for .bin files), reads tensor-cache.json for metadata, then downloads weight files in parallel using ProcessPoolExecutor.
7. Move to cache: Atomically moves the completed download from the temp directory to the cache location.

The parallel download uses a configurable number of worker processes (default 4):

with cf.ProcessPoolExecutor(max_workers=num_processes) as executor:
    futures = []
    for record in param_metadata:
        record_name = record["dataPath"]
        file_url = bin_url_template.format(user=user, repo=repo, record_name=record_name)
        file_dest = tmp_dir / record_name
        file_md5 = record.get("md5sum", None)
        futures.append(executor.submit(download_file, file_url, file_dest, file_md5))

get_or_download_model

def get_or_download_model(model: str) -> Path:

A high-level convenience function that resolves a model identifier to a local path:

1. If the model string starts with HF://, downloads via download_and_cache_mlc_weights.
2. Otherwise, treats it as a local path.
3. Validates that the path exists and contains mlc-chat-config.json.
4. Raises FileNotFoundError if the model directory or config file is missing.

Helper Functions

log_download_cache_policy

Logs the current value of MLC_DOWNLOAD_CACHE_POLICY to inform the user of the active policy.

_ensure_directory_not_exist

def _ensure_directory_not_exist(path: Path, force_redo: bool) -> None:

Ensures a directory does not already exist. If force_redo is True, removes any existing directory. Otherwise raises ValueError. Also creates parent directories as needed.

Dependencies

• concurrent.futures -- Parallel file downloading with ProcessPoolExecutor
• hashlib -- MD5 checksum verification
• json, os, shutil, subprocess, tempfile, pathlib.Path -- Standard library utilities
• requests -- HTTP file downloads
• mlc_llm.support.logging, mlc_llm.support.tqdm -- Internal logging and progress bars
• mlc_llm.support.constants -- Cache policy and home directory constants
• mlc_llm.support.style -- Terminal styling (bold)

File Location

python/mlc_llm/support/download_cache.py