Principle:Facebookresearch Audiocraft Training Checkpoint Resolution
Overview
Training Checkpoint Resolution is the process of locating and resolving model checkpoint files from either Dora experiment signatures or direct filesystem paths. In the AudioCraft training ecosystem, checkpoints are managed through Meta's Dora experiment management framework, which assigns unique hexadecimal signatures to each training run. Resolving these signatures to concrete checkpoint paths is the critical first step in any model export or deployment pipeline.
Theoretical Background
AudioCraft's training infrastructure uses Dora (a framework for running and managing experiments) to organize training runs. Each experiment is identified by a unique signature -- a short hexadecimal string derived from the experiment's configuration. The checkpoint resolution mechanism must handle two distinct addressing modes:
- Signature-based addressing: Using the
//sig/prefix convention, e.g.,//sig/abc123, which maps to a directory under Dora's experiment root (dora.dir/xps/abc123/). - Path-based addressing: Direct filesystem paths to checkpoint files or directories, which may need resolution through AudioCraft's reference path system.
The checkpoint naming convention follows a structured pattern: checkpoint_<name>.th, where name can be best for the best-performing checkpoint, a numeric epoch identifier, or omitted entirely for the latest checkpoint. For FSDP (Fully Sharded Data Parallel) training, rank-specific suffixes are appended (e.g., checkpoint_best.th.1 for rank 1).
Key Concepts
| Concept | Description |
|---|---|
| Dora Signature | A unique hexadecimal string identifying an experiment run, used to locate the experiment folder within the Dora XP root directory |
| Checkpoint Name Convention | Format checkpoint_<name>.th where name is "best", an epoch number, or empty for the latest checkpoint
|
| FSDP Sharding | In distributed training, each rank may have its own checkpoint shard, identified by a numeric suffix appended to the filename |
| Reference Path Resolution | AudioCraft's environment system can remap paths through AudioCraftEnvironment.resolve_reference_path() for portability across clusters
|
| XP Root Directory | The base directory for all Dora experiments, typically at dora.dir/xps/, under which each signature has its own subfolder
|
How It Works
The resolution process follows a decision tree:
- If the input starts with
//sig/, strip the prefix and look up the signature under the Dora XP root directory. - Otherwise, treat the input as a direct filesystem path and optionally resolve it through AudioCraft's reference path mechanism.
- If the resolved path is a directory rather than a file, append the standard checkpoint filename (e.g.,
checkpoint_best.th). - Finally, verify the path exists and return it, or return
Noneif no checkpoint is found.
This two-mode resolution allows scripts and notebooks to seamlessly reference checkpoints whether using Dora's experiment management system or direct file paths from custom training setups.
Design Rationale
The dual addressing scheme supports two primary user workflows:
- Researchers who use Dora for experiment tracking can reference checkpoints by their concise signatures, avoiding brittle hardcoded paths.
- Engineers deploying models can pass absolute paths directly, bypassing the Dora infrastructure entirely.
The optional FSDP awareness ensures correct checkpoint loading in distributed training environments where model parameters are sharded across multiple GPUs.