Implementation:Neuml Txtai Tokenizer

Knowledge Sources	Neuml_Txtai
Domains	NLP, Text Processing, Tokenization
Last Updated	2026-02-10 01:00 GMT

Overview

Concrete tool for text tokenization provided by txtai.

Description

Tokenizer is a pipeline that tokenizes text into a list of tokens using one of several configurable methods:

1. Unicode Text Segmentation (default): Splits using word boundary rules from the Unicode Standard Annex #29, similar to the standard tokenizer in Apache Lucene. Works well for most languages and supports emoji tokenization.
2. Alphanumeric filtering: Only accepts tokens that are at least 2 characters long with at least one non-trailing alphabetic character from the Latin alphabet. This is the backwards-compatible mode from older txtai versions.
3. Whitespace splitting: Simple tokenization on whitespace characters.
4. Regular expression: Tokenization using a user-provided regular expression pattern.

The pipeline supports optional lowercasing and stop word removal (with a built-in English stop word list matching Apache Lucene's defaults, or a user-provided list). A static tokenize() method provides a convenience function with backwards-compatible defaults (alphanumeric filtering and English stop words enabled).

Usage

Use Tokenizer when you need to split text into tokens for indexing, search, or text analysis. It is used internally by txtai for building search indexes and can be used directly for custom text processing. The Unicode segmentation mode is recommended for multilingual text, while the alphanumeric mode is suited for English-centric keyword extraction.

Code Reference

Source Location

• Repository: Neuml_Txtai
• File: src/python/txtai/pipeline/data/tokenizer.py

Signature

class Tokenizer(Pipeline):
    STOP_WORDS = {"a", "an", "and", "are", "as", "at", "be", "but", "by", ...}

    @staticmethod
    def tokenize(text, lowercase=True, emoji=True, alphanum=True,
                 stopwords=True, whitespace=False, regexp=None)

    def __init__(self, lowercase=True, emoji=True, alphanum=False,
                 stopwords=False, whitespace=False, regexp=None)
    def __call__(self, text)

Import

from txtai.pipeline.data.tokenizer import Tokenizer

I/O Contract

Inputs

Name	Type	Required	Description
text	str	Yes	Input text to tokenize.
lowercase	bool	No	Lowercases all tokens if True. Defaults to True.
emoji	bool	No	Includes emoji in tokenization if True. Defaults to True.
alphanum	bool	No	Requires 2+ character alphanumeric tokens if True. Defaults to False for __init__, True for static tokenize().
stopwords	bool or list	No	Removes English stop words if True, removes custom stop words if a list is provided. Defaults to False for __init__, True for static tokenize().
whitespace	bool	No	Tokenizes on whitespace if True. Defaults to False.
regexp	str	No	Regular expression pattern for tokenization. Defaults to None.

Outputs

Name	Type	Description
result	list of str	A list of token strings extracted from the input text. Returns None if input text is None.

Usage Examples

from txtai.pipeline import Tokenizer

# Static method with backwards-compatible defaults (alphanum + stopwords)
tokens = Tokenizer.tokenize("The quick brown fox jumps over the lazy dog")
# Returns: ['quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog']

# Create a tokenizer with Unicode segmentation (default instance settings)
tokenizer = Tokenizer()
tokens = tokenizer("Hello, world! This is a test.")
# Returns: ['hello', ',', 'world', '!', 'this', 'is', 'a', 'test', '.']

# Tokenize with stop word removal
tokenizer = Tokenizer(stopwords=True)
tokens = tokenizer("This is a test of the tokenizer")
# Returns: ['test', 'tokenizer']

# Alphanumeric tokenization
tokenizer = Tokenizer(alphanum=True, stopwords=True)
tokens = tokenizer("There are 3 cats and 2 dogs here!")
# Returns: ['cats', 'dogs', 'here']

# Whitespace tokenization
tokenizer = Tokenizer(whitespace=True)
tokens = tokenizer("simple whitespace splitting")
# Returns: ['simple', 'whitespace', 'splitting']

# Custom regular expression tokenization
tokenizer = Tokenizer(regexp=r"\w+")
tokens = tokenizer("hello-world foo_bar")

# Custom stop words list
tokenizer = Tokenizer(stopwords=["custom", "words"])
tokens = tokenizer("remove custom stop words from text")

Related Pages

• Environment:Neuml_Txtai_Python_Core_Dependencies