Implementation:Hpcaitech ColossalAI ChineseTextSplitter

Knowledge Sources	ColossalAI
Domains	RAG, NLP
Last Updated	2026-02-09 00:00 GMT

Overview

Chinese-aware text splitter extending LangChain's RecursiveCharacterTextSplitter for CJK text, provided by ColossalQA.

Description

ChineseTextSplitter adds Chinese punctuation marks as sentence boundary separators for more accurate splitting of Chinese text documents.

Usage

Use for Chinese or multilingual RAG applications. For English-only, use LangChain's standard RecursiveCharacterTextSplitter.

Code Reference

Source Location

• Repository: ColossalAI
• File: applications/ColossalQA/colossalqa/text_splitter/chinese_text_splitter.py
• Lines: 11-58

Signature

class ChineseTextSplitter(RecursiveCharacterTextSplitter):
    def __init__(
        self,
        separators: Optional[List[str]] = None,
        is_separator_regrx: bool = False,
        **kwargs: Any,
    ):
        """
        Default separators: ["\\n\\n", "\\n", "，", "。", "！", "？", "?"]
        Default chunk_size: 50
        Default chunk_overlap: 10
        """

    def split_text(self, text: str) -> List[str]:
        """Split text using Chinese-aware sentence boundaries."""

Import

from colossalqa.text_splitter.chinese_text_splitter import ChineseTextSplitter

I/O Contract

Inputs

Name	Type	Required	Description
text	str	Yes	Text to split
chunk_size	int	No	Max characters per chunk (default: 50)
chunk_overlap	int	No	Overlap between chunks (default: 10)

Outputs

Name	Type	Description
chunks	List[str]	List of text chunks respecting Chinese boundaries

Usage Examples

from colossalqa.text_splitter.chinese_text_splitter import ChineseTextSplitter

splitter = ChineseTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_text("这是一段很长的中文文本。它包含多个句子。")

Related Pages

Implements Principle

• Principle:Hpcaitech_ColossalAI_Text_Splitting_Chunking

Environment and Heuristic Links

• Environment:Hpcaitech_ColossalAI_ColossalQA_RAG_Environment