Implementation:Hpcaitech ColossalAI TableLoader

Knowledge Sources	Hpcaitech_ColossalAI
Domains	Data Loading, Database, NLP
Last Updated	2026-02-09 00:00 GMT

Overview

TableLoader is a data loader class that reads tabular data from various file formats and serves them as a SQLite database for SQL-based querying in ColossalQA pipelines.

Description

The TableLoader class supports loading data from multiple file formats including CSV, Excel, JSON, HTML, HDF5, Parquet, Feather, and Stata files using pandas. Upon initialization, it reads the specified files, converts them into pandas DataFrames, and stores them in a SQLite database via SQLAlchemy for downstream database operations. The class also handles cleanup of the SQL engine and data on deletion.

Usage

Use TableLoader when building a ColossalQA pipeline that needs to perform SQL-based question answering over structured tabular data. It is designed for scenarios where data resides in common tabular file formats and needs to be queried through a SQL interface.

Code Reference

Source Location

• Repository: Hpcaitech_ColossalAI
• File: applications/ColossalQA/colossalqa/data_loader/table_dataloader.py
• Lines: 1-115

Signature

class TableLoader:
    def __init__(self, files: str, sql_path: str = "sqlite:///mydatabase.db", verbose=False, **kwargs) -> None:
        ...

    def load_data(self, path):
        ...

    def to_sql(self, path, table_name):
        ...

    def get_sql_path(self):
        ...

    def __del__(self):
        ...

Import

from colossalqa.data_loader.table_dataloader import TableLoader

I/O Contract

Inputs

Name	Type	Required	Description
files	list[list[str, str]]	Yes	List of [file_path, dataset_name] pairs specifying the files to load and their corresponding table names
sql_path	str	No	SQLAlchemy connection string for the SQL database (default: "sqlite:///mydatabase.db")
verbose	bool	No	Whether to enable verbose logging output (default: False)
key	str	No	Key parameter for HDF5 file loading (passed via kwargs, default: "data")

Outputs

Name	Type	Description
get_sql_path return	str	The SQLAlchemy connection string for the created SQL database
to_sql return	str	The SQL path after loading data into the database

Supported File Formats

Extension	Format	Pandas Reader
.csv	Comma-Separated Values	pd.read_csv
.xlsx, .xls	Microsoft Excel	pd.read_excel
.json	JSON	pd.read_json
.html	HTML Tables	pd.read_html
.h5, .hdf5	HDF5	pd.read_hdf
.parquet	Apache Parquet	pd.read_parquet
.feather	Feather	pd.read_feather
.dta	Stata	pd.read_stata

Usage Examples

from colossalqa.data_loader.table_dataloader import TableLoader

# Load CSV and Excel files into a SQLite database
files = [
    ["/path/to/sales.csv", "sales_data"],
    ["/path/to/inventory.xlsx", "inventory"],
]

loader = TableLoader(
    files=files,
    sql_path="sqlite:///my_qa_database.db",
    verbose=True,
)

# Retrieve the SQL path for downstream use
sql_path = loader.get_sql_path()
print(sql_path)  # "sqlite:///my_qa_database.db"

Related Pages

• Environment:Hpcaitech_ColossalAI_ColossalQA_RAG_Environment