Implementation:Vespa engine Vespa FixedLengthChunker Chunk

Knowledge Sources	Vespa
Domains	NLP, Text_Processing
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for splitting long documents into fixed-length segments at word boundaries provided by Vespa's linguistics library. Supports configurable chunk length, CJK-aware boundary detection, and result caching.

Description

The FixedLengthChunker class implements the Chunker interface and provides a strategy for dividing input text into chunks of approximately equal length. The default chunk length is 1000 characters, but this can be overridden via context arguments.

The chunking algorithm is language-aware: for CJK (Chinese, Japanese, Korean) text, it respects character-level boundaries since CJK languages do not use spaces to delimit words. For Latin-script and other space-delimited languages, it snaps chunk boundaries to the nearest whitespace to avoid splitting words.

Key implementation characteristics:

• Configurable chunk length: The default is 1000 characters. A custom length can be passed as the first element of context.arguments().
• CJK detection: The method queries context.getLanguage().isCjk() to determine the appropriate boundary strategy.
• Result caching: The method uses context.computeCachedValueIfAbsent() with a composite cache key of (chunker instance, input text, chunk length, isCjk flag). This avoids redundant computation when the same text is chunked multiple times with the same parameters.
• Lazy computation: The actual chunking logic is encapsulated in an internal ChunkComputer class that is only invoked on a cache miss.

Usage

Use FixedLengthChunker.chunk() when you need to split documents into segments before embedding or indexing. Typical scenarios include:

• Preparing document text for embedding models that have maximum input length constraints.
• Creating chunk-level index entries for fine-grained retrieval.
• Processing documents of varying length into uniform-sized segments for batch operations.

Code Reference

Source Location

• Repository: Vespa
• File: linguistics/src/main/java/ai/vespa/language/chunker/FixedLengthChunker.java
• Lines: 33-53

Signature

@Override
public List<Chunk> chunk(String inputText, Context context)

Class Declaration

public class FixedLengthChunker implements Chunker

Package

package ai.vespa.language.chunker;

Imports

import com.yahoo.language.process.CharacterClasses;
import com.yahoo.language.process.Chunker;
import com.yahoo.text.UnicodeString;

Method Body

@Override
public List<Chunk> chunk(String inputText, Context context) {
    int chunkLength = context.arguments().isEmpty() ? defaultChunkLength : asInteger(context.arguments().get(0));
    boolean isCjk = context.getLanguage().isCjk();
    return context.computeCachedValueIfAbsent(new CacheKey(this, inputText, chunkLength, isCjk),
                                              () -> new ChunkComputer(inputText, chunkLength, isCjk).chunk());
}

I/O Contract

Inputs

Name	Type	Required	Description
inputText	String	Yes	The document text to split into chunks. Can be of any length.
context	Context	Yes	Processing context that provides: the detected language (via getLanguage()), optional arguments (first argument overrides chunk length), and a cache for storing computed results.

Context Arguments

Index	Type	Default	Description
0	Integer (as String)	1000	Maximum chunk length in characters. If not provided, the default chunk length of 1000 characters is used.

Outputs

Name	Type	Description
(return value)	List<Chunk>	An ordered list of chunks. Each Chunk contains a substring of the input text, split at appropriate boundaries. The chunks collectively cover the entire input text without gaps or overlaps.

Usage Examples

Basic Usage

import ai.vespa.language.chunker.FixedLengthChunker;
import com.yahoo.language.process.Chunker;
import com.yahoo.language.process.Chunker.Context;
import com.yahoo.language.process.Chunker.Chunk;
import java.util.List;

FixedLengthChunker chunker = new FixedLengthChunker();

// Create context with default language
Context context = new Context(Language.ENGLISH);

String longDocument = "A very long document text that needs to be split into chunks...";
List<Chunk> chunks = chunker.chunk(longDocument, context);

for (Chunk chunk : chunks) {
    System.out.println("Chunk: " + chunk.text());
}

Custom Chunk Length

import ai.vespa.language.chunker.FixedLengthChunker;
import com.yahoo.language.process.Chunker;
import com.yahoo.language.process.Chunker.Context;
import java.util.List;

FixedLengthChunker chunker = new FixedLengthChunker();

// Override chunk length to 500 characters via context arguments
Context context = new Context(Language.ENGLISH, List.of("500"));

List<Chunk> chunks = chunker.chunk(documentText, context);
// Each chunk will be approximately 500 characters, split at word boundaries

CJK Text Chunking

import ai.vespa.language.chunker.FixedLengthChunker;
import com.yahoo.language.Language;
import com.yahoo.language.process.Chunker;
import com.yahoo.language.process.Chunker.Context;
import java.util.List;

FixedLengthChunker chunker = new FixedLengthChunker();

// CJK language context - uses character-level boundary detection
Context cjkContext = new Context(Language.JAPANESE);

String japaneseText = "日本語のテキストを分割する必要がある場合...";
List<Chunk> chunks = chunker.chunk(japaneseText, cjkContext);
// Chunks split at character boundaries rather than whitespace

Related Pages

Implements Principle

• Principle:Vespa_engine_Vespa_Text_Chunking