Principle:Marker Inc Korea AutoRAG Corpus Instantiation
| Knowledge Sources | |
|---|---|
| Domains | NLP, Data_Modeling |
| Last Updated | 2026-02-08 06:00 GMT |
Overview
A data modeling pattern that wraps parsed documents and chunked passages into typed domain objects with fluent API chaining.
Description
Corpus Instantiation creates structured domain objects (Raw for parsed documents, Corpus for chunked passages) that carry both data and lineage information. The Raw class wraps a parsed DataFrame and can produce a Corpus via its chunk method. The Corpus class wraps a chunked DataFrame and links back to its source Raw. This linked chain enables the downstream QA generation pipeline to trace from generated questions back through passages to original documents. The fluent API design allows method chaining: Raw → chunk → Corpus → sample → QA → batch_apply → filter → to_parquet.
Usage
Use this principle after document chunking when you need to construct typed data objects for the QA generation pipeline. It is the bridge between raw data processing (parse/chunk) and the evaluation dataset creation workflow.
Theoretical Basis
The pattern is based on the concept of linked data containers with immutable lineage:
- A Raw object holds parsed document data and can produce Corpus objects via chunking
- A Corpus object holds chunked passages and maintains a read-only link to its source Raw
- A QA object holds question-answer pairs and maintains a read-only link to its source Corpus
This ensures that at any point in the pipeline, you can trace a generated QA pair back to its source passage and original document.
# Abstract data flow pattern
raw = Raw(parsed_df) # Wrap parsed data
corpus = raw.chunk("sentence") # Chunk into passages (linked)
qa = corpus.sample(sampler, n=100) # Sample passages for QA (linked)
# qa.linked_corpus → corpus → corpus.linked_raw → raw