Implementation:Explodinggradients Ragas Tokenizers Module

Field	Value
source	Explodinggradients_Ragas (GitHub)
domains	NLP, Tokenization
last_updated	2026-02-10 00:00 GMT

Overview

The tokenizers module provides a unified tokenizer interface with implementations for tiktoken (OpenAI) and HuggingFace tokenizers, plus a factory function and lazy-initialized default tokenizer.

Description

BaseTokenizer is an abstract base class defining encode (text to token IDs), decode (token IDs to text), and count_tokens methods. TiktokenWrapper wraps tiktoken encodings, accepting a pre-initialized encoding, model name (e.g., "gpt-4"), or encoding name (e.g., "o200k_base"), defaulting to "o200k_base". HuggingFaceTokenizer wraps HuggingFace tokenizers, accepting a pre-initialized tokenizer or model name, with a lazy import of transformers. The get_tokenizer factory function creates instances by type string ("tiktoken" or "huggingface"). A _LazyTokenizer class provides a lazily-initialized default tokenizer exposed as the module-level DEFAULT_TOKENIZER constant, avoiding network calls at import time. The get_default_tokenizer function creates the default singleton on first access.

Usage

Use get_tokenizer() to create a tokenizer for a specific backend and model. Use DEFAULT_TOKENIZER for quick access to the default tiktoken tokenizer without explicit initialization.

Code Reference

Item	Detail
Source Location	src/ragas/tokenizers.py L16-200
Classes	BaseTokenizer(ABC), TiktokenWrapper, HuggingFaceTokenizer, _LazyTokenizer
Functions	get_tokenizer(tokenizer_type, model_name, encoding_name) -> BaseTokenizer, get_default_tokenizer() -> TiktokenWrapper
Import	from ragas.tokenizers import get_tokenizer, TiktokenWrapper, HuggingFaceTokenizer

I/O Contract

Inputs (get_tokenizer)

Parameter	Type	Description
tokenizer_type	str	"tiktoken" or "huggingface"
model_name	Optional[str]	Model name (e.g., "gpt-4", "meta-llama/Llama-2-7b")
encoding_name	Optional[str]	Tiktoken encoding name (e.g., "cl100k_base", "o200k_base")

Outputs

Method	Return Type	Description
get_tokenizer()	BaseTokenizer	Configured tokenizer instance
encode(text)	List[int]	Token ID list for the input text
decode(tokens)	str	Decoded text from token IDs
count_tokens(text)	int	Number of tokens in the text

Usage Examples

from ragas.tokenizers import get_tokenizer, TiktokenWrapper, DEFAULT_TOKENIZER

# Default tiktoken tokenizer (lazy init)
count = DEFAULT_TOKENIZER.count_tokens("Hello, world!")
print(count)  # e.g., 4

# Create tiktoken tokenizer for a specific model
tokenizer = get_tokenizer("tiktoken", model_name="gpt-4")
tokens = tokenizer.encode("Hello, world!")
print(tokens)             # [9906, 11, 1917, 0]
print(tokenizer.decode(tokens))  # "Hello, world!"

# Create HuggingFace tokenizer
hf_tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b")
print(hf_tokenizer.count_tokens("Hello, world!"))

# Direct TiktokenWrapper usage
wrapper = TiktokenWrapper(encoding_name="cl100k_base")
print(wrapper.count_tokens("Testing token count"))

Related Pages

• Utils_Module - num_tokens_from_string uses tokenizers
• Validation_Module - Framework validation utilities