Implementation:Run llama Llama index TokenCounter

Overview

The TokenCounter class provides token counting utilities for strings, chat messages, and tool definitions. It estimates token usage for OpenAI-compatible message formats, accounting for special tokens added by the chat protocol (e.g., per-message overhead, function call tokens, role tokens). The implementation is adapted from the openai-token-counter library.

Source File: llama-index-core/llama_index/core/utilities/token_counting.py (103 lines)

Module: llama_index.core.utilities.token_counting

Dependencies

Module	Import
llama_index.core.base.llms.types	ChatMessage, MessageRole
llama_index.core.utils	get_tokenizer

Class Definition

class TokenCounter:
    """
    Token counter class.

    Attributes:
        model (Optional[str]): The model to use for token counting.
    """

Constructor

def __init__(self, tokenizer: Optional[Callable[[str], list]] = None) -> None

Parameter	Type	Default	Description
tokenizer	Optional[Callable[[str], list]]	None	A callable that takes a string and returns a list of tokens. Defaults to get_tokenizer() which provides a standard tokenizer.

The tokenizer is stored as self.tokenizer and used by all counting methods.

Core Methods

get_string_tokens

def get_string_tokens(self, string: str) -> int

Returns the token count for a plain string by calling the tokenizer and returning the length of the resulting list.

estimate_tokens_in_messages

def estimate_tokens_in_messages(self, messages: List[ChatMessage]) -> int

Estimates the total token count for a list of chat messages. For each message, the method accounts for:

Component	Token Calculation
Role	Tokenizes the role string (e.g., "user", "assistant", "system")
Content	Tokenizes the message content (only if it is a string)
Function call (legacy)	Tokenizes function name and arguments separately, adds 3 overhead tokens
Tool calls	For each tool call with a function attribute, tokenizes the function name and arguments, adds 3 overhead tokens per call
Per-message overhead	Adds 3 tokens per message
Function/tool role adjustment	Subtracts 2 tokens if the message role is FUNCTION or TOOL

The method handles both the legacy function_call format and the newer tool_calls format found in additional_kwargs.

Token overhead constants:

• +3 per message: Standard overhead for message framing
• +3 per function/tool call: Overhead for function call structure
• -2 for function/tool role: Adjustment for messages with function or tool roles

estimate_tokens_in_tools

def estimate_tokens_in_tools(self, tools: List[Dict[str, Any]]) -> int

Estimates the token count for a list of tool definitions (as produced by to_openai_tool() or similar). The method converts the entire tool list to a string representation and tokenizes it. Returns 0 if the tools list is empty.

Token Counting Flow

For each message:
  tokens += tokenize(role)
  tokens += tokenize(content)      [if string]
  tokens += tokenize(func.name)    [if function_call]
  tokens += tokenize(func.args)    [if function_call]
  tokens += 3                      [if function_call]
  for each tool_call:
    tokens += tokenize(tool.function.name)
    tokens += tokenize(tool.function.arguments)
    tokens += 3                    [per tool call]
  tokens += 3                      [per message overhead]
  tokens -= 2                      [if role is FUNCTION or TOOL]

Design Notes

• The token counting logic follows OpenAI's chat completion token counting conventions, where each message incurs overhead tokens for framing.
• The function_call handling in additional_kwargs is maintained for backward compatibility with older OpenAI API formats. The newer tool_calls format is also supported.
• Tool calls are accessed via hasattr(tool_call, "function"), indicating they may be either dict-like or object-like depending on the API version.
• The estimate_tokens_in_tools method uses a simple str() serialization approach, which provides a rough estimate rather than an exact count. This is acceptable because tool definitions are typically static and the exact count matters less than the message count.
• The tokenizer is injectable via the constructor, allowing different tokenizers for different models (e.g., tiktoken for GPT models, sentencepiece for others).

See Also

• SQLDatabase -- Another utility class in the same package
• Core Types -- Core type definitions including TokenGen and TokenAsyncGen