Principle:Mlfoundations Open flamingo Optimizer And Scheduler Configuration

Overview

Training optimization strategy combining AdamW with decoupled weight decay and a warmup-then-decay learning rate schedule for stable convergence of vision-language models.

Description

The optimizer configuration for OpenFlamingo training uses AdamW with parameter group separation: gated cross-attention parameters receive weight decay while other trainable parameters do not. The learning rate schedule supports three variants: constant with warmup, linear decay with warmup, and cosine decay with warmup. Gradient accumulation enables effective batch sizes larger than GPU memory allows.

Usage

When configuring the training loop for OpenFlamingo; after model creation and distributed wrapping.

Theoretical Basis

AdamW decouples weight decay from gradient updates, applying L2 regularization directly to parameter values rather than through the gradient. Warmup prevents unstable early training by gradually increasing the learning rate. Cosine annealing provides smooth decay to a near-zero learning rate. Parameter group separation allows different regularization for different module types (cross-attention vs perceiver).

Decoupled weight decay: Weight decay is applied as a direct subtraction from parameter values (theta = theta - wd * theta), not as an L2 penalty in the loss.
Warmup phase: Learning rate ramps linearly from 0 to the target value over a configurable number of steps, preventing gradient explosion in early training.
Cosine annealing: After warmup, the learning rate follows a cosine curve from the peak value down to near zero, providing a smooth and theoretically motivated decay.
Parameter group separation: Gated cross-attention layers receive weight decay to regularize newly initialized parameters, while pretrained or non-attention parameters are left unregularized.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment