Principle:FlagOpen FlagEmbedding LLM Representation Learning
| Knowledge Sources | |
|---|---|
| Domains | Machine Learning, Large Language Models, Representation Learning, Embedding Models |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Adapting large language models as text embedding models through contrastive pretraining and efficient fine-tuning techniques like LoRA to create high-quality embeddings from decoder-only architectures.
Description
This principle addresses the challenge of transforming causal language models (like Llama) into effective bidirectional encoders for text embedding tasks. Traditional LLMs use causal attention masks that prevent tokens from attending to future positions, which is suboptimal for embedding generation. The LLARA/Llama2Vec approach introduces techniques to enable bidirectional attention through pretraining with masked language modeling objectives, followed by contrastive fine-tuning on retrieval tasks. The method uses parameter-efficient adaptation (LoRA) to avoid full model retraining. This enables leveraging powerful pretrained LLMs for embedding tasks while maintaining their semantic knowledge.
Usage
Use this principle when:
- Converting decoder-only LLMs into embedding models
- Leveraging pretrained language models for retrieval tasks
- Building embedding models that benefit from LLM reasoning capabilities
- Creating parameter-efficient adaptations of large models for downstream tasks
Theoretical Basis
The LLM representation learning pipeline consists of:
- Bidirectional Pretraining:
- Replace causal mask with bidirectional attention
- Train with masked LM objective: L_MLM = -Σ log P(x_masked | context)
- Use LoRA adapters: W = W_0 + BA where B ∈ R^(d×r), A ∈ R^(r×k), r << d
- Contrastive Fine-tuning:
- Extract embeddings from last token or mean pooling
- Apply contrastive loss: L = -log(exp(sim(q, d+)/τ) / Σ_i exp(sim(q, d_i)/τ))
- Use instruction prefixes to distinguish queries vs documents
- Efficient Adaptation:
- Freeze base LLM parameters W_0
- Only train LoRA matrices A, B
- Reduces trainable parameters by 99%+ while maintaining performance
The key insight is that LLMs already contain rich semantic representations; enabling bidirectional attention and task-specific adaptation unlocks their embedding potential without full retraining.