Implementation:LLMBook zh LLMBook zh github io Trainer Train SFT

Knowledge Sources	LLMBook-zh HuggingFace Trainer
Domains	NLP, Training
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for executing the SFT training loop with loss masking using HuggingFace Trainer provided by the Transformers library.

Description

In the SFT context, Trainer.train() executes the training loop with SFTDataset (providing loss-masked labels) and DataCollatorForSupervisedDataset (handling variable-length padding). After training, save_model and save_state persist the fine-tuned model.

This is a Wrapper Doc — it documents how the LLMBook repository uses HuggingFace Trainer for supervised fine-tuning specifically.

Usage

Call trainer.train() after setting up the Trainer with SFT components.

Code Reference

Source Location

Repository: LLMBook-zh
File: code/7.1 SFT实践.py
Lines: 64-89

Signature

trainer = Trainer(
    model=model,                              # AutoModelForCausalLM
    args=args,                                # Arguments(TrainingArguments)
    tokenizer=tokenizer,                      # AutoTokenizer
    train_dataset=SFTDataset(args, tokenizer),# SFT dataset with loss masking
    data_collator=DataCollatorForSupervisedDataset(tokenizer),
)
trainer.train()
trainer.save_model(args.output_dir + "/checkpoint-final")
trainer.save_state()

Import

from transformers import Trainer, TrainingArguments, HfArgumentParser

I/O Contract

Inputs

Name	Type	Required	Description
model	PreTrainedModel	Yes	Base model for fine-tuning
args	TrainingArguments	Yes	Training hyperparameters
tokenizer	AutoTokenizer	Yes	Tokenizer
train_dataset	SFTDataset	Yes	SFT dataset with masked labels
data_collator	DataCollatorForSupervisedDataset	Yes	Padding collator

Outputs

Name	Type	Description
train() returns	TrainOutput	Training metrics
checkpoint	Files	Saved model at output_dir/checkpoint-final

Usage Examples

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer
from dataset.sft_dataset import SFTDataset

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    attn_implementation="flash_attention_2"
)

trainer = Trainer(
    model=model,
    args=args,
    tokenizer=tokenizer,
    train_dataset=SFTDataset(args, tokenizer),
    data_collator=DataCollatorForSupervisedDataset(tokenizer),
)
trainer.train()
trainer.save_model(args.output_dir + "/checkpoint-final")

Related Pages

Implements Principle

Principle:LLMBook_zh_LLMBook_zh_github_io_SFT_Training_Execution

Requires Environment

Uses Heuristic

Heuristic:LLMBook_zh_LLMBook_zh_github_io_BF16_Mixed_Precision_Default

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment