Implementation:Guardrails ai Guardrails Docs Utils

Knowledge Sources	Guardrails
Domains	Utilities, Text Processing
Last Updated	2026-02-14 00:00 GMT

Overview

Provides utility functions and classes for document processing, including text splitting, sentence tokenization, PDF reading, and chunk generation.

Description

The Docs Utils module contains several utilities for handling text and document-based operations within Guardrails:

• messages_to_prompt_string -- Converts a list of message dictionaries (containing Prompt, Instructions, or plain strings) into a single concatenated prompt string. It extracts the .source attribute from Prompt/Instructions objects.

• TextSplitter -- A class that splits text into chunks with token-level boundaries using the GPT-2 tokenizer from tiktoken. It supports configurable chunk sizes, overlap, buffer sizes, and prompt template awareness (subtracting prompt template tokens from the available chunk budget).

• sentence_split -- A standalone function that splits text into sentences using NLTK's sent_tokenize, with automatic download of the punkt tokenizer if not already present.

• read_pdf -- Reads a PDF file at the given path and extracts its text content page by page using the pypdfium2 library, returning a single string with newline separators.

• get_chunks_from_text -- A flexible chunking function supporting multiple strategies: "sentence", "word", "char", "token", and "full". Each strategy breaks text into atomic units and then groups them into overlapping chunks of the specified size.

The module has optional dependencies on tiktoken and nltk, which are imported gracefully with fallback handling.

Usage

Use this module when you need to preprocess documents for validation or embedding pipelines. The TextSplitter class is suited for token-aware chunking when working with LLM prompt budgets. Use get_chunks_from_text for flexible chunking strategies, and read_pdf for extracting text from PDF documents.

Code Reference

Source Location

• Repository: Guardrails
• File: guardrails/utils/docs_utils.py

Signature

def messages_to_prompt_string(
    messages: Union[
        list[dict[str, Union[str, Prompt, Instructions]]], MessageHistory
    ],
) -> str: ...

class TextSplitter:
    def __init__(self) -> None: ...
    def split(
        self,
        text: str,
        tokens_per_chunk: int = 2048,
        token_overlap: int = 512,
        buffer: int = 128,
        prompt_template: Optional[Prompt] = None,
    ) -> List[str]: ...
    def prompt_template_token_length(self, prompt_template: Prompt) -> int: ...
    def __call__(self, *args, **kwds) -> Any: ...

def sentence_split(text: str) -> List[str]: ...

def read_pdf(path) -> str: ...

def get_chunks_from_text(
    text: str,
    chunk_strategy: str,
    chunk_size: int,
    chunk_overlap: int,
) -> List[str]: ...

Import

from guardrails.utils.docs_utils import (
    messages_to_prompt_string,
    TextSplitter,
    sentence_split,
    read_pdf,
    get_chunks_from_text,
)

I/O Contract

messages_to_prompt_string

Parameter	Type	Description
messages	Union[list[dict[str, Union[str, Prompt, Instructions]]], MessageHistory]	A list of message dicts with "content" keys

Returns: str -- All message contents concatenated into a single string.

TextSplitter.split

Parameter	Type	Default	Description
text	str	required	The text to split into chunks
tokens_per_chunk	int	2048	Maximum tokens per chunk
token_overlap	int	512	Number of overlapping tokens between consecutive chunks
buffer	int	128	Token buffer subtracted from chunk size
prompt_template	Optional[Prompt]	None	If provided, its token length is subtracted from the chunk budget

Returns: List[str] -- List of text chunks with token-level boundaries.

sentence_split

Parameter	Type	Description
text	str	The text to split into sentences

Returns: List[str] -- A list of sentence strings.

read_pdf

Parameter	Type	Description
path	Any	File path to the PDF document

Returns: str -- Extracted text content from the PDF with \r characters removed.

get_chunks_from_text

Parameter	Type	Description
text	str	The text to chunk
chunk_strategy	str	Strategy: "sentence", "word", "char", "token", or "full"
chunk_size	int	Number of atomic units per chunk
chunk_overlap	int	Number of atomic units to overlap between chunks

Returns: List[str] -- List of text chunks.

Raises:

• ImportError -- If nltk or tiktoken is not installed for strategies that require them
• ValueError -- If chunk_strategy is not one of the recognized strategies

Usage Examples

from guardrails.utils.docs_utils import TextSplitter

# Split text into token-bounded chunks
splitter = TextSplitter()
chunks = splitter.split(
    text="A very long document text...",
    tokens_per_chunk=1024,
    token_overlap=256,
)
for chunk in chunks:
    print(chunk[:80], "...")

from guardrails.utils.docs_utils import get_chunks_from_text

# Chunk text by sentences with overlap
chunks = get_chunks_from_text(
    text="First sentence. Second sentence. Third sentence. Fourth sentence.",
    chunk_strategy="sentence",
    chunk_size=2,
    chunk_overlap=1,
)
# Result: ["First sentence. Second sentence.", "Second sentence. Third sentence.", ...]

from guardrails.utils.docs_utils import read_pdf

# Extract text from a PDF file
text = read_pdf("/path/to/document.pdf")
print(text[:200])

Related Pages

• Guardrails_ai_Guardrails_VectorDBBase -- May use chunked text as input for vector embeddings
• Guardrails_ai_Guardrails_Prompt -- The Prompt class used by TextSplitter and messages_to_prompt_string