Implementation:Run llama Llama index SentenceTransformersFinetuneEngine Init

Overview

SentenceTransformersFinetuneEngine is the primary class for finetuning Sentence Transformer embedding models in LlamaIndex. The __init__ method configures the training pipeline: loading the pretrained model, preparing training data as InputExample pairs, setting up the loss function, creating the data loader, and optionally configuring validation evaluation and checkpointing.

Source Location

Property	Value
File	llama-index-finetuning/llama_index/finetuning/embeddings/sentence_transformer.py
Lines	15-109
Class	SentenceTransformersFinetuneEngine
Base Class	BaseEmbeddingFinetuneEngine
Import	from llama_index.finetuning import SentenceTransformersFinetuneEngine

Constructor Signature

class SentenceTransformersFinetuneEngine(BaseEmbeddingFinetuneEngine):
    def __init__(
        self,
        dataset: EmbeddingQAFinetuneDataset,
        model_id: str = "BAAI/bge-small-en",
        model_output_path: str = "exp_finetune",
        batch_size: int = 10,
        val_dataset: Optional[EmbeddingQAFinetuneDataset] = None,
        loss: Optional[Any] = None,
        epochs: int = 2,
        show_progress_bar: bool = True,
        evaluation_steps: int = 50,
        use_all_docs: bool = False,
        trust_remote_code: bool = False,
        device: Optional[Any] = None,
        save_checkpoints: bool = False,
        resume_from_checkpoint: bool = False,
        checkpoint_save_steps: int = 500,
        checkpoint_save_total_limit: int = 0,
    ) -> None:

Parameters

Parameter	Type	Default	Description
dataset	EmbeddingQAFinetuneDataset	required	Training dataset containing queries, corpus, and relevance mappings.
model_id	str	"BAAI/bge-small-en"	HuggingFace model identifier for the pretrained Sentence Transformer.
model_output_path	str	"exp_finetune"	Directory path where the finetuned model will be saved.
batch_size	int	10	Number of training examples per batch. Larger batches yield more in-batch negatives.
val_dataset	Optional[EmbeddingQAFinetuneDataset]	None	Optional validation dataset for evaluation during training.
loss	Optional[Any]	None	Custom loss function. If None, defaults to MultipleNegativesRankingLoss.
epochs	int	2	Number of training epochs.
show_progress_bar	bool	True	Whether to display a progress bar during training.
evaluation_steps	int	50	Evaluate on validation set every N training steps.
use_all_docs	bool	False	If True, creates pairs for all relevant docs per query; if False, uses only the first.
trust_remote_code	bool	False	Allow execution of remote code in model loading (HuggingFace models).
device	Optional[Any]	None	PyTorch device for model placement (e.g., "cuda", "cpu").
save_checkpoints	bool	False	Enable periodic checkpoint saving during training.
resume_from_checkpoint	bool	False	Resume training from the latest checkpoint.
checkpoint_save_steps	int	500	Save a checkpoint every N training steps.
checkpoint_save_total_limit	int	0	Maximum number of checkpoints to retain (0 = unlimited).

Internal Initialization Sequence

The constructor performs the following steps:

1. Load the Sentence Transformer model from HuggingFace:

   self.model = SentenceTransformer(model_id, trust_remote_code=trust_remote_code, device=device)

1. Prepare training examples from the dataset:

examples = []
for query_id, query in dataset.queries.items():
    if use_all_docs:
        for node_id in dataset.relevant_docs[query_id]:
            text = dataset.corpus[node_id]
            example = InputExample(texts=[query, text])
            examples.append(example)
    else:
        node_id = dataset.relevant_docs[query_id][0]
        text = dataset.corpus[node_id]
        example = InputExample(texts=[query, text])
        examples.append(example)

1. Create the DataLoader with the specified batch size:

   self.loader = DataLoader(examples, batch_size=batch_size)

1. Set up the evaluator (if validation data provided):

if val_dataset is not None:
    evaluator = InformationRetrievalEvaluator(
        val_dataset.queries, val_dataset.corpus, val_dataset.relevant_docs
    )

1. Configure the loss function (default: MultipleNegativesRankingLoss):

   self.loss = loss or losses.MultipleNegativesRankingLoss(self.model)

1. Calculate warmup steps as 10% of total training steps:

   self.warmup_steps = int(len(self.loader) * epochs * 0.1)

1. Configure checkpoint path (if checkpointing enabled):

   self.checkpoint_path = os.path.join("checkpoints", model_output_path) if save_checkpoints else None

Usage Example

from llama_index.finetuning import (
    SentenceTransformersFinetuneEngine,
    EmbeddingQAFinetuneDataset,
)

# Load datasets
train_dataset = EmbeddingQAFinetuneDataset.from_json("train_dataset.json")
val_dataset = EmbeddingQAFinetuneDataset.from_json("val_dataset.json")

# Configure finetuning engine
finetune_engine = SentenceTransformersFinetuneEngine(
    dataset=train_dataset,
    model_id="BAAI/bge-small-en",
    model_output_path="finetuned_model",
    val_dataset=val_dataset,
    batch_size=16,
    epochs=3,
    evaluation_steps=100,
    save_checkpoints=True,
    checkpoint_save_steps=200,
)

Dependencies

• sentence_transformers.SentenceTransformer -- Pretrained model loading and training
• sentence_transformers.InputExample -- Training example format
• sentence_transformers.losses.MultipleNegativesRankingLoss -- Default loss function
• sentence_transformers.evaluation.InformationRetrievalEvaluator -- Validation evaluator
• torch.utils.data.DataLoader -- Batch data loading

Knowledge Sources

LlamaIndex Finetuning Source Sentence Transformers API

Metadata

Machine Learning Embeddings Finetuning LlamaIndex

Principle:Run_llama_Llama_index_Embedding_Finetune_Configuration Environment:Run_llama_Llama_index_Sentence_Transformers_Finetuning Heuristic:Run_llama_Llama_index_Finetuning_Warmup_Steps