Principle:Marker Inc Korea AutoRAG Corpus Ingestion
| Knowledge Sources | |
|---|---|
| Domains | Information_Retrieval, Indexing |
| Last Updated | 2026-02-08 06:00 GMT |
Overview
An indexing technique that loads corpus passages into retrieval backends (BM25 lexical index and vector databases) for search.
Description
Corpus Ingestion is the process of pre-computing retrieval indexes from the corpus data before running the evaluation pipeline. Two types of ingestion are supported: BM25 lexical ingestion tokenizes passages and stores them as a pickle file for fast BM25 retrieval, and vector database ingestion computes embedding vectors and stores them in a vector store (Chroma, Milvus, Pinecone, Qdrant, Weaviate, or Couchbase). Both methods support incremental ingestion, skipping documents that already exist in the index.
Usage
This principle is automatically applied at the start of an evaluation trial if the pipeline config includes BM25 or VectorDB retrieval nodes. It ensures retrieval indexes are ready before any retrieval module is evaluated.
Theoretical Basis
BM25 Ingestion:
- Tokenize corpus passages using a configurable tokenizer (Porter stemmer, Korean Kiwi, Japanese Sudachi, or HuggingFace tokenizer)
- Store tokenized passages and doc_ids as a pickle file for BM25 scoring at query time
Vector DB Ingestion:
- Compute embedding vectors for each passage using an embedding model (OpenAI, HuggingFace)
- Store vectors with doc_ids in the configured vector database
- API-based embedding (OpenAI) uses async batch ingestion
- Local model embedding (HuggingFace) uses synchronous batch encoding