Principle:Zai org CogVideo Checkpointing and Validation
| Principle Metadata | |
|---|---|
| Name | Checkpointing_and_Validation |
| Category | Training |
| Domains | Fine_Tuning, Diffusion_Models |
| Knowledge Sources | CogVideo Repository, CogVideoX Paper |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
Checkpointing and Validation is the principle of periodic model state saving and quality assessment during training to enable recovery and monitor convergence.
Description
Checkpointing saves the full training state (model weights, optimizer states, learning rate scheduler state) at regular intervals to enable resumption after interruptions. Validation generates sample videos from held-out prompts to visually assess training progress. Together they provide fault tolerance and quality feedback during long training runs.
The checkpointing system includes:
- Full state saving: Model weights, optimizer states, scheduler states, global step counter, and random number generator states are saved using
accelerator.save_state(). - Safe serialization: Weights are saved in
.safetensorsformat for security and compatibility. - Rolling checkpoint limit: Only the most recent N checkpoints are retained to prevent disk exhaustion (default: 10).
- Resumption: Training can be resumed from any checkpoint by loading the full state and continuing from the saved step.
The validation system includes:
- Inference pipeline construction: During validation, a full inference pipeline is temporarily constructed from the current model state.
- Sample generation: Videos are generated from held-out validation prompts using the current LoRA-adapted model.
- Video export: Generated videos are exported to files and optionally logged to experiment trackers (wandb).
Usage
Use during any CogVideoX fine-tuning run. Checkpointing is essential for multi-day training runs where interruptions are likely. Validation is optional but recommended for monitoring quality and detecting issues like mode collapse or training divergence.
Theoretical Basis
Checkpoint frequency trades off storage cost versus recovery granularity. For CogVideoX fine-tuning:
- A typical LoRA checkpoint is ~50-200 MB (only adapter weights).
- A full training state checkpoint includes optimizer states and can be several GB.
- The default interval of 200 steps with a limit of 10 rolling checkpoints balances recovery granularity with storage cost.
Validation on held-out prompts provides qualitative assessment of training progress. Unlike image generation, video quality metrics (FID, FVD) are computationally expensive and require large sample sizes. Visual inspection of generated videos remains the most practical approach for monitoring fine-tuning quality. Key indicators to watch during validation:
- Motion quality: Videos should exhibit smooth, coherent motion.
- Text alignment: Generated content should match the validation prompts.
- Visual fidelity: Resolution, color, and detail should be maintained.
- Temporal consistency: Objects and scenes should remain consistent across frames.
Rolling checkpoint limits (e.g., keep last 10) prevent disk exhaustion during long training runs while maintaining enough history for recovery and comparison.