Principle:Run llama Llama index Docstore Deduplication
| Knowledge Sources | |
|---|---|
| Domains | Data_Ingestion, RAG, Data_Management |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
Document deduplication ensures that re-ingesting documents into a RAG pipeline does not create duplicate entries in the vector store or docstore, using content hashing and document ID tracking.
Description
When running ingestion pipelines repeatedly (e.g., on a schedule or as new documents arrive), the same document may be processed multiple times. Without deduplication, this leads to:
- Duplicate vectors in the vector store, wasting storage and skewing retrieval results
- Redundant processing of already-ingested content, wasting compute
- Stale data remaining in the store when source documents are deleted
LlamaIndex's docstore-based deduplication addresses these problems by maintaining a record of previously ingested documents. Each document is identified by a hash of its content, and the ingestion pipeline compares incoming documents against this record to determine which ones need processing.
Usage
Enable deduplication by providing a docstore to the IngestionPipeline and selecting a DocstoreStrategy. The strategy determines how duplicates and changes are handled.
Theoretical Basis
Deduplication Strategies
LlamaIndex provides three deduplication strategies, each suited to different operational requirements:
- UPSERTS (default): Insert new documents and update existing ones whose content has changed. Documents removed from the source are not deleted from the store. This is the safest strategy for append-only workflows.
- DUPLICATES_ONLY: Only skip documents that already exist with identical content. Changed documents are re-inserted as new entries rather than updating existing ones. Useful when you want to preserve historical versions.
- UPSERTS_AND_DELETE: Like UPSERTS, but also deletes documents from the store that are no longer present in the incoming batch. This ensures the store is an exact mirror of the source, but requires the full document set to be provided on each run.
Content Hashing
Documents are identified by computing a hash of their content. When a document with the same ID but different content hash is encountered, the pipeline knows the document has been modified and needs re-processing:
# Conceptual deduplication flow
for document in incoming_documents:
existing = docstore.get(document.doc_id)
if existing is None:
# New document - process and store
process(document)
elif existing.hash != document.hash:
# Changed document - re-process and update
update(document)
else:
# Identical document - skip
pass