Principle:Langgenius Dify Document Processing Configuration

Knowledge Sources	Dify Dify Knowledge Base Docs Chunking Strategies for LLM Applications
Domains	RAG Document Chunking Text Processing
Last Updated	2026-02-08 00:00 GMT

Overview

Document processing configuration defines the chunking and preprocessing strategies that control how ingested documents are split into indexable segments for optimal retrieval quality.

Description

After raw data enters a knowledge base, it must be transformed into discrete, semantically meaningful chunks before embedding and indexing can occur. Document processing configuration governs every aspect of this transformation:

• Pre-processing rules -- automated cleanup steps such as removing extra whitespace, stripping URLs or email addresses, and normalizing encoding.
• Segmentation strategy -- the algorithm and parameters that determine where chunk boundaries fall, including separator characters, maximum segment length, and overlap between adjacent segments.
• Processing mode -- a high-level selector that bundles these settings into predefined profiles (automatic, custom, or hierarchical).

The quality of retrieval in a RAG system is highly sensitive to chunking decisions. Segments that are too large dilute relevance signals and waste context window tokens; segments that are too small lose coherence and force the retriever to assemble meaning from scattered fragments. The processing configuration is therefore one of the most impactful tuning surfaces in the entire pipeline.

Usage

Configure document processing when:

• Creating a new knowledge base -- the creation wizard prompts the user to choose a processing mode and, if custom is selected, to set specific segmentation parameters.
• Re-processing existing documents -- if retrieval quality is poor, an operator may adjust chunk size, overlap, or separators and re-index the affected documents.
• Switching to hierarchical chunking -- for long-form documents with clear heading structures, hierarchical mode can produce a tree of parent and child segments that improves both precision and context retention.

Theoretical Basis

Chunking Strategies

Automatic Mode

Automatic mode applies a platform-default configuration that balances generality and performance:

mode: "automatic"
rules:
  pre_processing_rules:
    - id: remove_extra_spaces,      enabled: true
    - id: remove_urls_emails,       enabled: false
  segmentation:
    separator: "\n"
    max_tokens: 500
    chunk_overlap: 50

This mode is suitable for most document types and requires no user intervention.

Custom Mode

Custom mode exposes the full set of parameters, allowing fine-grained control:

Parameter	Description	Typical Range
separator	Character or string used to split text before enforcing token limits	\n, \n\n, 。, custom regex
max_tokens	Maximum number of tokens per segment	100 -- 2000
chunk_overlap	Number of tokens shared between adjacent segments to preserve context continuity	0 -- 200
remove_extra_spaces	Collapse runs of whitespace into a single space	true / false
remove_urls_emails	Strip URLs and email addresses from text	true / false

Hierarchical Mode

Hierarchical chunking respects document structure (headings, sections, subsections) to produce a tree of segments:

Document
  ├── Section A  (parent segment)
  │     ├── Paragraph 1  (child segment)
  │     └── Paragraph 2  (child segment)
  └── Section B  (parent segment)
        └── Paragraph 3  (child segment)

At retrieval time, matching a child segment can cause its parent segment to be included in the context window, providing broader context without sacrificing precision.

The Overlap Problem

Overlap is a trade-off between context continuity and index bloat:

• With zero overlap, a sentence that straddles a chunk boundary is split across two segments and may not be retrievable by either.
• With high overlap, the same tokens appear in multiple segments, increasing storage and embedding costs and potentially returning near-duplicate results.

A common heuristic is to set overlap to 10--20% of the maximum segment size.

Pre-Processing Pipeline

Pre-processing rules run before segmentation:

Raw Text
  → Remove extra whitespace  (optional)
  → Remove URLs / emails     (optional)
  → Normalize encoding        (always)
  → Segmentation algorithm
  → Segment[]

Each rule is independently toggleable, giving operators control over how aggressively the text is cleaned.

Related Pages

Implemented By

• Implementation:Langgenius_Dify_FetchProcessRule