Implementation:LLMBook zh LLMBook zh github io AutoPeftModelForCausalLM Merge And Unload

Knowledge Sources	LLMBook-zh PEFT Library
Domains	Deep_Learning, Parameter_Efficient_Finetuning, Deployment
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for loading LoRA checkpoints and merging adapters into the base model using the PEFT library, as used in the LLMBook repository.

Description

AutoPeftModelForCausalLM.from_pretrained loads a saved LoRA checkpoint, and merge_and_unload() merges adapter weights into the base model, producing a standalone PreTrainedModel. The repository iterates over all checkpoint subdirectories and creates merged versions for each.

This is a Wrapper Doc documenting how the LLMBook repository uses the PEFT library for adapter merging.

The implementation also handles DeepSpeed ZeRO-3 compatibility by calling unset_hf_deepspeed_config() before merging.

Usage

Use this after LoRA training to produce standalone merged model checkpoints for deployment.

Code Reference

Source Location

Repository: LLMBook-zh
File: code/7.4 LoRA实践.py
Lines: 45-56

Signature

# Load LoRA checkpoint
peft_model = AutoPeftModelForCausalLM.from_pretrained(checkpoint_path: str) -> PeftModel

# Merge adapters into base model
merged_model = peft_model.merge_and_unload() -> PreTrainedModel

# Save standalone model
merged_model.save_pretrained(save_path: str)
tokenizer.save_pretrained(save_path: str)

Import

from peft import AutoPeftModelForCausalLM
from transformers.integrations.deepspeed import is_deepspeed_zero3_enabled, unset_hf_deepspeed_config

External Reference

PEFT Documentation

I/O Contract

Inputs

Name	Type	Required	Description
checkpoint_path	str	Yes	Path to saved LoRA checkpoint directory

Outputs

Name	Type	Description
merged_model	PreTrainedModel	Standalone model with merged LoRA weights
saved files	Files	Model weights + tokenizer at {checkpoint}-merged/

Usage Examples

import os
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

output_dir = "./lora_output"
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Iterate over all checkpoints and merge
for subdir in os.listdir(output_dir):
    if subdir.startswith("checkpoint"):
        checkpoint_path = os.path.join(output_dir, subdir)
        peft_model = AutoPeftModelForCausalLM.from_pretrained(checkpoint_path)
        merged_model = peft_model.merge_and_unload()

        save_path = checkpoint_path + "-merged"
        merged_model.save_pretrained(save_path)
        tokenizer.save_pretrained(save_path)
        print(f"Merged model saved to {save_path}")

Related Pages

Implements Principle

Principle:LLMBook_zh_LLMBook_zh_github_io_LoRA_Adapter_Merging

Requires Environment

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment