Heuristic:Run llama Llama index Finetuning Warmup Steps
| Knowledge Sources | |
|---|---|
| Domains | Embedding_Finetuning, Optimization |
| Last Updated | 2026-02-11 19:00 GMT |
Overview
Warmup steps and hyperparameter defaults for LlamaIndex embedding adapter finetuning: 10% warmup, AdamW optimizer, 2e-5 learning rate.
Description
The `EmbeddingAdapterFinetuneEngine` automatically calculates warmup steps as 10% of total training steps (data loader size x epochs x 0.1). It uses AdamW optimizer with a learning rate of 2e-5 and gradient clipping at 1.0. The similarity loss uses a scale factor of 20.0. These defaults are shared across the adapter, sentence transformer, and cross-encoder finetuning engines.
Usage
Apply this heuristic when running Embedding Finetuning with the adapter engine or sentence transformer engine. Relevant when:
- Adjusting training hyperparameters for convergence
- The default warmup is too aggressive or too conservative
- Training on very small or very large datasets
The Insight (Rule of Thumb)
- Action (Warmup): Warmup steps = `len(data_loader) * epochs * 0.1` (10% of total steps).
- Action (Learning Rate): Default AdamW lr = `2e-5`. Conservative but stable.
- Action (Gradient Clipping): `max_grad_norm = 1.0` prevents gradient explosion.
- Action (Loss Scale): `scale = 20.0` for MultipleNegativesRankingLoss. Higher = sharper similarity distribution.
- Action (Batch Size): Default batch_size = 10 for both adapter and sentence transformer engines.
- Trade-off: Higher learning rate = faster convergence but risk of overshooting. Lower warmup fraction = aggressive start, may cause instability on small datasets.
Reasoning
10% Warmup: This is a widely-adopted heuristic in the NLP/deep learning community. Warmup gradually increases the learning rate from near-zero, preventing large gradient updates early in training when the model has random weights. 10% provides a balance between stability and not wasting too many training steps on warmup.
AdamW at 2e-5: This learning rate is standard for finetuning pre-trained models. It is intentionally much lower than typical training-from-scratch rates (1e-3 to 1e-4) because the pre-trained weights are already in a good region of the loss landscape.
Dimension Auto-Detection: The adapter engine uses a hack to auto-detect embedding dimensions by embedding the string "hello world". This avoids requiring users to know the dimension of their embedding model.
Code evidence from `embeddings/adapter.py:88`:
self._warmup_steps = int(len(self.loader) * epochs * 0.1)
Default hyperparameters from `embeddings/adapter_utils.py:55-61`:
warmup_steps: int = 10000,
optimizer_class: Type[Optimizer] = torch.optim.AdamW,
optimizer_params: Dict[str, Any] = {"lr": 2e-5},
max_grad_norm: float = 1,
Loss scale from `embeddings/adapter_utils.py:28-34`:
scale: float = 20.0,
Dimension hack from `embeddings/adapter.py:62-65`:
# HACK: get dimension by passing text through it
if dim is None:
test_embedding = self.embed_model.get_text_embedding("hello world")
self.dim = len(test_embedding)