Implementation:Togethercomputer Together python Text Preprocessing Pattern

Overview

The Text Preprocessing Pattern implements the Principle:Togethercomputer_Together_python_Text_Preprocessing principle by providing user-defined preprocessing logic for cleaning and preparing text before calling the Together Python SDK's embedding and reranking APIs.

This is a Pattern Doc -- the Together Python SDK does not provide built-in preprocessing utilities. Instead, this documents the recommended patterns for preparing text inputs on the user side.

Pattern Structure

The preprocessing pattern follows three stages:

1. Clean -- Remove noise (HTML, special characters, excessive whitespace) from raw text
2. Normalize -- Standardize text format (casing, Unicode, whitespace)
3. Chunk/Truncate -- Split or truncate text to fit within model token limits

Example Patterns

Basic Text Cleaning

import re

def clean_text(text: str) -> str:
    """Remove HTML tags, normalize whitespace, and strip control characters."""
    # Remove HTML tags
    text = re.sub(r"<[^>]+>", "", text)
    # Remove control characters
    text = re.sub(r"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]", "", text)
    # Normalize whitespace (collapse multiple spaces/newlines)
    text = re.sub(r"\s+", " ", text).strip()
    return text


# Usage before embedding
from together import Together

client = Together()
raw_texts = ["<p>Hello <b>world</b></p>", "  Multiple   spaces  here  "]
cleaned = [clean_text(t) for t in raw_texts]

response = client.embeddings.create(
    input=cleaned,
    model="togethercomputer/m2-bert-80M-8k-retrieval",
)

Document Chunk Splitting

def chunk_text(text: str, max_chars: int = 2000, overlap: int = 200) -> list[str]:
    """Split text into overlapping chunks by character count.

    Args:
        text: The input text to split.
        max_chars: Maximum characters per chunk (approximate token limit proxy).
        overlap: Number of overlapping characters between adjacent chunks.

    Returns:
        List of text chunks.
    """
    if len(text) <= max_chars:
        return [text]

    chunks = []
    start = 0
    while start < len(text):
        end = start + max_chars
        # Try to break at a sentence boundary
        if end < len(text):
            last_period = text.rfind(".", start, end)
            if last_period > start + max_chars // 2:
                end = last_period + 1
        chunks.append(text[start:end].strip())
        start = end - overlap
    return chunks


# Usage: chunk a long document before embedding
long_document = "..." # A long document
chunks = chunk_text(long_document, max_chars=2000, overlap=200)

response = client.embeddings.create(
    input=chunks,
    model="togethercomputer/m2-bert-80M-8k-retrieval",
)

Preprocessing for Reranking

def prepare_for_reranking(documents: list[str], max_chars: int = 4000) -> list[str]:
    """Prepare documents for reranking: clean and truncate.

    For reranking, documents should remain as coherent units (not chunked),
    but should be cleaned and truncated to fit model limits.
    """
    prepared = []
    for doc in documents:
        # Clean but preserve paragraph structure
        doc = re.sub(r"<[^>]+>", "", doc)
        doc = re.sub(r"\n{3,}", "\n\n", doc)
        doc = doc.strip()
        # Truncate if too long (preserve leading content)
        if len(doc) > max_chars:
            # Try to truncate at a sentence boundary
            truncated = doc[:max_chars]
            last_period = truncated.rfind(".")
            if last_period > max_chars // 2:
                truncated = truncated[:last_period + 1]
            doc = truncated
        prepared.append(doc)
    return prepared


# Usage before reranking
query = "What is retrieval-augmented generation?"
raw_docs = ["<p>RAG combines retrieval...</p>", "Very long document..." * 500]
cleaned_docs = prepare_for_reranking(raw_docs)

response = client.rerank.create(
    model="Salesforce/Llama-Rank-V1",
    query=query,
    documents=cleaned_docs,
    top_n=5,
)

Batch Deduplication

def deduplicate_texts(texts: list[str]) -> tuple[list[str], dict[int, int]]:
    """Remove duplicate texts, returning unique texts and index mapping.

    Returns:
        unique_texts: Deduplicated list of texts.
        index_map: Maps original indices to unique text indices.
    """
    seen = {}
    unique_texts = []
    index_map = {}

    for i, text in enumerate(texts):
        normalized = text.strip().lower()
        if normalized not in seen:
            seen[normalized] = len(unique_texts)
            unique_texts.append(text)
        index_map[i] = seen[normalized]

    return unique_texts, index_map


# Usage: deduplicate before embedding to save API calls
texts = ["Hello world", "hello world", "Unique text", "Hello world"]
unique, mapping = deduplicate_texts(texts)
# unique = ["Hello world", "Unique text"], mapping = {0: 0, 1: 0, 2: 1, 3: 0}

response = client.embeddings.create(
    input=unique,
    model="togethercomputer/m2-bert-80M-8k-retrieval",
)

# Reconstruct full embedding list using the mapping
all_embeddings = [response.data[mapping[i]].embedding for i in range(len(texts))]

Design Decisions

Decision	Recommendation	Rationale
Character-based vs. token-based chunking	Character-based as a proxy, or use a tokenizer library	Character-based is simpler but less precise; token-based requires a tokenizer matching the target model
Overlap between chunks	10-20% overlap recommended	Prevents loss of context at chunk boundaries; improves retrieval recall
Chunking vs. truncation for reranking	Truncation preferred	Reranking cross-encoders benefit from coherent documents, not fragments
Preprocessing order	Clean -> Normalize -> Chunk	Cleaning first removes noise that could affect chunk boundaries

Source Files

This is a user-side pattern. No SDK source files implement this functionality. The relevant SDK entry points that receive preprocessed text are:

• src/together/resources/embeddings.py -- Embeddings.create()
• src/together/resources/rerank.py -- Rerank.create()

Metadata

Property	Value
Implementation	Text Preprocessing Pattern
Type	Pattern Doc (user-defined logic)
Domain	NLP, Information_Retrieval, RAG
Workflow	Embeddings_And_Reranking
Principle	Principle:Togethercomputer_Together_python_Text_Preprocessing

Knowledge Sources

• together-python
• Together API Docs

Related Pages

• Principle:Togethercomputer_Together_python_Text_Preprocessing