Principle:Langgenius Dify Document Processing Configuration
| Knowledge Sources | |
|---|---|
| Domains | RAG Document Chunking Text Processing |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Document processing configuration defines the chunking and preprocessing strategies that control how ingested documents are split into indexable segments for optimal retrieval quality.
Description
After raw data enters a knowledge base, it must be transformed into discrete, semantically meaningful chunks before embedding and indexing can occur. Document processing configuration governs every aspect of this transformation:
- Pre-processing rules -- automated cleanup steps such as removing extra whitespace, stripping URLs or email addresses, and normalizing encoding.
- Segmentation strategy -- the algorithm and parameters that determine where chunk boundaries fall, including separator characters, maximum segment length, and overlap between adjacent segments.
- Processing mode -- a high-level selector that bundles these settings into predefined profiles (automatic, custom, or hierarchical).
The quality of retrieval in a RAG system is highly sensitive to chunking decisions. Segments that are too large dilute relevance signals and waste context window tokens; segments that are too small lose coherence and force the retriever to assemble meaning from scattered fragments. The processing configuration is therefore one of the most impactful tuning surfaces in the entire pipeline.
Usage
Configure document processing when:
- Creating a new knowledge base -- the creation wizard prompts the user to choose a processing mode and, if custom is selected, to set specific segmentation parameters.
- Re-processing existing documents -- if retrieval quality is poor, an operator may adjust chunk size, overlap, or separators and re-index the affected documents.
- Switching to hierarchical chunking -- for long-form documents with clear heading structures, hierarchical mode can produce a tree of parent and child segments that improves both precision and context retention.
Theoretical Basis
Chunking Strategies
Automatic Mode
Automatic mode applies a platform-default configuration that balances generality and performance:
mode: "automatic"
rules:
pre_processing_rules:
- id: remove_extra_spaces, enabled: true
- id: remove_urls_emails, enabled: false
segmentation:
separator: "\n"
max_tokens: 500
chunk_overlap: 50
This mode is suitable for most document types and requires no user intervention.
Custom Mode
Custom mode exposes the full set of parameters, allowing fine-grained control:
| Parameter | Description | Typical Range |
|---|---|---|
| separator | Character or string used to split text before enforcing token limits | \n, \n\n, 。, custom regex
|
| max_tokens | Maximum number of tokens per segment | 100 -- 2000 |
| chunk_overlap | Number of tokens shared between adjacent segments to preserve context continuity | 0 -- 200 |
| remove_extra_spaces | Collapse runs of whitespace into a single space | true / false |
| remove_urls_emails | Strip URLs and email addresses from text | true / false |
Hierarchical Mode
Hierarchical chunking respects document structure (headings, sections, subsections) to produce a tree of segments:
Document
├── Section A (parent segment)
│ ├── Paragraph 1 (child segment)
│ └── Paragraph 2 (child segment)
└── Section B (parent segment)
└── Paragraph 3 (child segment)
At retrieval time, matching a child segment can cause its parent segment to be included in the context window, providing broader context without sacrificing precision.
The Overlap Problem
Overlap is a trade-off between context continuity and index bloat:
- With zero overlap, a sentence that straddles a chunk boundary is split across two segments and may not be retrievable by either.
- With high overlap, the same tokens appear in multiple segments, increasing storage and embedding costs and potentially returning near-duplicate results.
A common heuristic is to set overlap to 10--20% of the maximum segment size.
Pre-Processing Pipeline
Pre-processing rules run before segmentation:
Raw Text
→ Remove extra whitespace (optional)
→ Remove URLs / emails (optional)
→ Normalize encoding (always)
→ Segmentation algorithm
→ Segment[]
Each rule is independently toggleable, giving operators control over how aggressively the text is cleaned.