Implementation:NVIDIA NeMo Aligner Load From NeMo

Implementation Metadata
Name	Load_From_NeMo
Type	API Doc
Implements Principle	Pretrained_Model_Loading
Repository	NeMo Aligner
File	nemo_aligner/utils/utils.py
Lines	L78-152
Domains	NLP, Transfer_Learning
Last Updated	2026-02-07 00:00 GMT

Overview

Concrete tool for loading pretrained NeMo GPT models from checkpoint archives provided by the NeMo Aligner utility module.

Description

The load_from_nemo function restores a pretrained model from a .nemo checkpoint file or extracted directory. It first optionally loads and modifies the checkpoint configuration, then uses NeMo's restore_from mechanism with a CustomSaveRestoreConnector to instantiate the model class with the merged configuration and restored weights. The companion function load_and_override_model_config handles loading the checkpoint's embedded config and merging it with task-specific overrides using OmegaConf.

Usage

Import these functions when initializing any alignment training script. Called at the start of every training pipeline (SFT, RM, DPO, PPO, REINFORCE) to load the pretrained model that will be fine-tuned.

Code Reference

Source Location

Repository: NeMo Aligner
File: nemo_aligner/utils/utils.py
Lines: L78-152

Signature

def load_from_nemo(
    cls,
    model_cfg,
    trainer,
    strict=True,
    modify_config_fn=None,
    restore_path=None,
    load_base_model_only=False,
    return_updated_cfg=False,
):
    """load a model using nemo checkpoint"""

def load_and_override_model_config(
    restore_path: str,
    model_cfg_to_overwrite: DictConfig,
    remove_meta_info: bool = True,
) -> DictConfig:
    """load the config in the model checkpoint and then overwrite it"""

Import

from nemo_aligner.utils.utils import load_from_nemo, load_and_override_model_config

I/O Contract

Inputs

Name	Type	Required	Description
cls	Type[Model]	Yes	Model class to instantiate (e.g., GPTSFTModel, MegatronGPTDPOModel)
model_cfg	DictConfig	Yes	Merged model configuration
trainer	pytorch_lightning.Trainer	Yes	PTL trainer instance
restore_path	str	Yes	Path to .nemo checkpoint file or extracted directory
strict	bool	No	Whether to strictly match state dict keys (default True)
modify_config_fn	Callable	No	Optional function to further modify config
load_base_model_only	bool	No	Load only base model, skip adapter weights
return_updated_cfg	bool	No	Also return the updated config alongside the model

Outputs

Name	Type	Description
model	Model	Instantiated model with restored weights
model_cfg	DictConfig	(Optional) Updated configuration if return_updated_cfg=True

Usage Examples

Loading for SFT

from nemo_aligner.utils.utils import load_from_nemo, load_and_override_model_config
from nemo_aligner.models.nlp.gpt.gpt_sft_model import GPTSFTModel

# 1. Load and merge config
model_cfg = load_and_override_model_config(
    restore_path="/models/gpt-43b.nemo",
    model_cfg_to_overwrite=cfg.model,
)

# 2. Instantiate model with merged config
model = load_from_nemo(
    GPTSFTModel,
    model_cfg,
    trainer,
    strict=True,
    restore_path=cfg.model.restore_from_path,
)

Related Pages

Knowledge Sources

NeMo Aligner

NLP | Transfer_Learning

2026-02-07 00:00 GMT

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment