Principle:Eric mitchell Direct preference optimization Model Loading
| Knowledge Sources | |
|---|---|
| Domains | Model_Initialization, NLP, Deep_Learning |
| Last Updated | 2026-02-08 02:00 GMT |
Overview
A technique for initializing pre-trained causal language models from checkpoint weights with configurable precision and device placement strategies.
Description
Model loading is the process of instantiating a pre-trained language model architecture and populating it with saved weights. For DPO training, this involves loading a HuggingFace causal language model with specific dtype configuration and device mapping. The loaded model serves as the starting point for either SFT training (single model) or DPO training (policy + reference model pair).
Key considerations include:
- Dtype selection: Models can be loaded in float32, float16, or bfloat16 depending on GPU support and training requirements
- Device placement: BasicTrainer uses device_map="balanced" to naively split across GPUs; FSDP and TensorParallel trainers load to CPU first and shard later
- Memory efficiency: The low_cpu_mem_usage flag reduces peak CPU memory during loading
- Dropout disabling: After loading, dropout is disabled to ensure deterministic training behavior
Usage
Use this principle at the beginning of any training pipeline when you need to initialize models from pre-trained HuggingFace checkpoints. For DPO, two copies of the model are loaded: one as the trainable policy and one as the frozen reference.
Theoretical Basis
Pre-trained language model weights encode learned representations from large-scale unsupervised training. Loading these weights provides a strong initialization that captures language understanding, which is then refined through supervised fine-tuning (SFT) or preference optimization (DPO).
The choice of numerical precision affects both memory usage and training dynamics:
- float32: Full precision; most numerically stable but uses 4 bytes per parameter
- float16/bfloat16: Half precision; halves memory but may require careful loss scaling
Pseudo-code:
# Abstract model loading (NOT actual implementation)
model = load_pretrained(model_name, dtype=target_dtype, device_map=strategy)
disable_dropout(model) # ensure deterministic behavior