Implementation:Hiyouga LLaMA Factory Processor Utils

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Data Processing, Algorithms
Last Updated	2026-02-06 19:00 GMT

Overview

Provides the abstract DatasetProcessor base class and utility functions for sequence length inference and greedy knapsack packing used by all concrete dataset processors.

Description

This module defines the foundational DatasetProcessor abstract base class, which holds references to the template, tokenizer, processor, and data arguments, and declares the preprocess_dataset and print_data_example abstract methods that all concrete processors must implement. It also provides three critical utility functions: infer_seqlen intelligently distributes a cutoff length budget between source (prompt) and target (response) sequences, prioritizing the shorter component; search_for_fit performs binary search to find the largest number fitting a given capacity; and greedy_knapsack implements an efficient bin-packing algorithm that groups variable-length sequences into fixed-capacity bins for sequence packing during training.

Usage

Use DatasetProcessor as the base class when implementing a new dataset processor. Use infer_seqlen whenever you need to truncate prompt-response pairs to fit within a maximum sequence length. Use greedy_knapsack when implementing packed training to efficiently group multiple shorter examples into fixed-length batches.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/data/processor/processor_utils.py
• Lines: 1-88

Signature

@dataclass
class DatasetProcessor(ABC):
    template: "Template"
    tokenizer: "PreTrainedTokenizer"
    processor: Optional["ProcessorMixin"]
    data_args: "DataArguments"

    @abstractmethod
    def preprocess_dataset(self, examples: dict[str, list[Any]]) -> dict[str, list[Any]]: ...

    @abstractmethod
    def print_data_example(self, example: dict[str, list[int]]) -> None: ...

def search_for_fit(numbers: list[int], capacity: int) -> int

def greedy_knapsack(numbers: list[int], capacity: int) -> list[list[int]]

def infer_seqlen(source_len: int, target_len: int, cutoff_len: int) -> tuple[int, int]

Import

from llamafactory.data.processor.processor_utils import DatasetProcessor, infer_seqlen, greedy_knapsack, search_for_fit

I/O Contract

Inputs

Name	Type	Required	Description
source_len	int	Yes	Length of the source (prompt) token sequence (for infer_seqlen)
target_len	int	Yes	Length of the target (response) token sequence (for infer_seqlen)
cutoff_len	int	Yes	Maximum allowed total sequence length (for infer_seqlen)
numbers	list[int]	Yes	List of sequence lengths to pack (for greedy_knapsack)
capacity	int	Yes	Maximum bin capacity (for greedy_knapsack)

Outputs

Name	Type	Description
(new_source_len, new_target_len)	tuple[int, int]	Truncated source and target lengths that fit within cutoff_len (from infer_seqlen)
knapsacks	list[list[int]]	Groups of sequence lengths packed into bins (from greedy_knapsack)
index	int	Index of the largest fitting number, or -1 if none fits (from search_for_fit)

Usage Examples

from llamafactory.data.processor.processor_utils import infer_seqlen, greedy_knapsack

# Distribute a cutoff budget of 2048 between prompt and response
source_len, target_len = infer_seqlen(source_len=500, target_len=2000, cutoff_len=2048)
# Result: source is truncated minimally, target gets most of the budget

# Pack sequences of varying lengths into bins of capacity 4096
lengths = [1024, 2048, 512, 3000, 1500, 800]
bins = greedy_knapsack(lengths, capacity=4096)
# Each bin in bins is a list of lengths that sum to <= 4096

Related Pages

• Hiyouga_LLaMA_Factory_Supervised_Processor - Concrete processor extending DatasetProcessor for SFT, uses infer_seqlen and greedy_knapsack
• Hiyouga_LLaMA_Factory_Pairwise_Processor - Concrete processor extending DatasetProcessor for pairwise preference training
• Hiyouga_LLaMA_Factory_Feedback_Processor - Concrete processor extending DatasetProcessor for KTO training
• Hiyouga_LLaMA_Factory_Data_Args - DataArguments providing the cutoff_len parameter