Implementation:Hpcaitech ColossalAI DocumentLoader

Knowledge Sources	ColossalAI
Domains	RAG, Data_Engineering
Last Updated	2026-02-09 00:00 GMT

Overview

Multi-format document loader wrapping LangChain loaders for the ColossalQA RAG pipeline.

Description

DocumentLoader dispatches file loading to format-specific LangChain loaders based on file extension. It accumulates all loaded documents into a flat list for downstream text splitting and embedding.

Usage

Create with a list of file paths, which triggers automatic loading of all documents.

Code Reference

Source Location

• Repository: ColossalAI
• File: applications/ColossalQA/colossalqa/data_loader/document_loader.py
• Lines: 23-137

Signature

class DocumentLoader:
    def __init__(self, files: List, **kwargs) -> None:
        """
        Args:
            files: List of file paths or [path, name] pairs
            **kwargs: Keyword args for loaders (jq_schema, content_key for JSON)
        """

    def load_data(self, path: str) -> None:
        """Load a single file, dispatching to the correct format loader."""

Import

from colossalqa.data_loader.document_loader import DocumentLoader

I/O Contract

Inputs

Name	Type	Required	Description
files	List	Yes	File paths or [path, name] pairs
jq_schema	str	No	JQ filter for JSON files (default: ".data[]")
content_key	str	No	JSON content field (default: "content")

Outputs

Name	Type	Description
all_data	List[Document]	Flat list of LangChain Document objects

Usage Examples

from colossalqa.data_loader.document_loader import DocumentLoader

loader = DocumentLoader(
    files=[
        ["/docs/manual.pdf", "Product Manual"],
        ["/docs/faq.txt", "FAQ"],
        ["/docs/data.csv", "Data"],
    ]
)

print(f"Loaded {len(loader.all_data)} documents")

Related Pages

Implements Principle

• Principle:Hpcaitech_ColossalAI_Document_Loading

Environment and Heuristic Links

• Environment:Hpcaitech_ColossalAI_ColossalQA_RAG_Environment