Principle:Intel Ipex llm Data Preparation For Finetuning
| Knowledge Sources | |
|---|---|
| Domains | NLP, Data_Processing |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Methodology for transforming raw instruction-following datasets into tokenized training samples using prompt templates.
Description
Data Preparation for Finetuning encompasses the pipeline of loading raw datasets (from HuggingFace Hub or local files), applying prompt templates to format instruction-input-output triples, tokenizing with truncation and padding, optionally masking input tokens from the loss computation, and splitting into train/validation sets. The Alpaca prompt template format is the most common pattern, supporting both instruction-only and instruction+input variants.
Usage
Use this principle when preparing instruction-following datasets for supervised fine-tuning (SFT) of language models. It is applicable to LoRA, QLoRA, and full fine-tuning workflows where the training data follows the instruction/input/output format (e.g., Alpaca, Dolly, or similar datasets).
Theoretical Basis
The data preparation pipeline follows these steps:
# Abstract algorithm (NOT real implementation)
1. Load dataset from HuggingFace Hub or local JSON/JSONL
2. Initialize Prompter with template (e.g., "alpaca")
3. For each data point:
a. Format using template: prompt = template.format(instruction, input, output)
b. Tokenize with truncation at cutoff_len
c. If not train_on_inputs: mask input portion with label=-100
d. Append EOS token if needed
4. Split into train/val sets (val_set_size samples for validation)