Implementation:Hiyouga LLaMA Factory MCA Workflow

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Distributed Training, Megatron-Core
Last Updated	2026-02-06 19:00 GMT

Overview

Provides Megatron-Core Adapter (MCA) training workflows for pre-training, supervised fine-tuning, and DPO with large-scale distributed parallelism.

Description

The mca/workflow module implements three training workflows using the mcore_adapter library for Megatron-style distributed training. run_pt handles pre-training with McaTrainer and DataCollatorForSeq2Seq. run_sft handles supervised fine-tuning with SFTDataCollatorWith4DAttentionMask, supporting sequence packing and optional freezing of vision tower, multi-modal projector, or language model layers for Qwen VL models. run_dpo handles Direct Preference Optimization using McaDPOTrainer with a configurable DPOConfig for loss type, beta, and label smoothing. All workflows use a _data_collator_wrapper that adjusts labels and input_ids by shifting one position to match MCA's internal shift logic (removing the last input token and first label token). The module validates model support against MCA_SUPPORTED_MODELS and supports expert model parallelism with max-length padding when expert_model_parallel_size is greater than 1.

Usage

Use these workflows when training with the Megatron-Core backend for large-scale distributed training with tensor, pipeline, and expert parallelism. This provides an alternative to the standard HuggingFace Trainer-based workflows, enabling training at scales requiring Megatron-style parallelism. Requires the mcore_adapter package to be installed.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/train/mca/workflow.py
• Lines: 1-291

Signature

def _data_collator_wrapper(data_collator: Any) -> Callable

def _check_model_support(model_args: "ModelArguments") -> None

def run_pt(
    model_args: "ModelArguments",
    data_args: "DataArguments",
    training_args: "McaSeq2SeqTrainingArguments",
    finetuning_args: "FinetuningArguments",
    callbacks: Optional[list["TrainerCallback"]] = None,
) -> None

def run_sft(
    model_args: "ModelArguments",
    data_args: "DataArguments",
    training_args: "McaSeq2SeqTrainingArguments",
    finetuning_args: "FinetuningArguments",
    callbacks: Optional[list["TrainerCallback"]] = None,
) -> None

def run_dpo(
    model_args: "ModelArguments",
    data_args: "DataArguments",
    training_args: "McaSeq2SeqTrainingArguments",
    finetuning_args: "FinetuningArguments",
    callbacks: Optional[list["TrainerCallback"]] = None,
) -> None

Import

from llamafactory.train.mca.workflow import run_pt, run_sft, run_dpo

I/O Contract

Inputs

Name	Type	Required	Description
model_args	ModelArguments	Yes	Model loading configuration including model_name_or_path and trust_remote_code
data_args	DataArguments	Yes	Dataset configuration including cutoff_len (adjusted +1/-1 for MCA shift), neat_packing, packing settings
training_args	McaSeq2SeqTrainingArguments	Yes	MCA-specific training arguments including expert_model_parallel_size, sequence_packing, resume_from_checkpoint
finetuning_args	FinetuningArguments	Yes	Contains freeze_vision_tower, freeze_multi_modal_projector, freeze_language_model, pref_beta, pref_loss, dpo_label_smoothing, plot_loss, include_effective_tokens_per_second
callbacks	Optional[list[TrainerCallback]]	No	Additional trainer callbacks; SaveProcessorCallback is added automatically when a processor is present

Outputs

Name	Type	Description
None	None	Side effects: saves model, metrics, and trainer state to output_dir; generates loss plots if configured

Usage Examples

# Running MCA-based SFT training
from llamafactory.train.mca.workflow import run_sft

run_sft(
    model_args=model_args,
    data_args=data_args,
    training_args=mca_training_args,
    finetuning_args=finetuning_args,
)

Related Pages

• Hiyouga_LLaMA_Factory_DPO_Workflow - Standard HuggingFace Trainer-based DPO workflow
• Hiyouga_LLaMA_Factory_KTO_Workflow - KTO preference training workflow
• Hiyouga_LLaMA_Factory_Model_Patcher - Model patching used in standard workflows (not MCA)