Implementation:Run llama Llama index Playground

Overview

The Playground module provides an experimentation harness for comparing the performance of different LlamaIndex indices, retriever modes, and configurations against user queries. It executes queries across multiple index and retriever mode combinations, collects timing and token usage metrics, and presents results in a structured format. This module is located at llama-index-core/llama_index/core/playground/base.py (190 lines).

Purpose

The Playground enables rapid prototyping and evaluation of different index strategies. Users can pass the same query to multiple indices (e.g., VectorStoreIndex, TreeIndex, SummaryIndex) with various retriever modes and compare outputs, response times, and token consumption side by side. This is valuable for tuning retrieval strategies and understanding the trade-offs between different index types.

Constants

Name	Type	Description
DEFAULT_INDEX_CLASSES	List[Type[BaseIndex]]	Default list of index classes used when creating indices from documents: [VectorStoreIndex, TreeIndex, SummaryIndex].
DEFAULT_MODES	Dict[Type[BaseIndex], List[str]]	Default retriever modes for each index type. TreeIndex uses all TreeRetrieverMode values, SummaryIndex uses all ListRetrieverMode values, and VectorStoreIndex uses ["default"].

Type Alias

INDEX_SPECIFIC_QUERY_MODES_TYPE = Dict[Type[BaseIndex], List[str]]

Maps index classes to their list of retriever mode strings.

Key Components

Class: Playground

The main class for running comparative experiments across indices and retriever modes.

Constructor

def __init__(
    self,
    indices: List[BaseIndex],
    retriever_modes: INDEX_SPECIFIC_QUERY_MODES_TYPE = DEFAULT_MODES,
)

Parameter	Type	Description
indices	List[BaseIndex]	A list of pre-built index instances to experiment with. Must be non-empty.
retriever_modes	INDEX_SPECIFIC_QUERY_MODES_TYPE	A mapping from index types to lists of retriever mode strings. Defaults to DEFAULT_MODES.

The constructor validates both inputs, then initializes a color mapping for display purposes.

Class Method: from_docs

@classmethod
def from_docs(
    cls,
    documents: List[Document],
    index_classes: List[Type[BaseIndex]] = DEFAULT_INDEX_CLASSES,
    retriever_modes: INDEX_SPECIFIC_QUERY_MODES_TYPE = DEFAULT_MODES,
    **kwargs: Any,
) -> Playground

Factory method that creates a Playground from a list of documents. Builds one index per class in index_classes using index_class.from_documents(). Raises ValueError if the document list is empty.

Properties

Property	Type	Description
indices	List[BaseIndex]	Getter/setter for the list of indices. Setter validates the input.
retriever_modes	dict	Getter/setter for the retriever modes mapping. Setter validates the input.

Validation Methods

Method	Description
_validate_indices	Ensures the indices list is non-empty and every element is a BaseIndex instance.
_validate_modes	Ensures the retriever modes dictionary is non-empty.

Method: compare

def compare(
    self,
    query_text: str,
    to_pandas: bool | None = True,
) -> Any | List[Dict[str, Any]]

The core experimentation method that runs the query across all index/mode combinations.

Parameters:

Parameter	Type	Description
query_text	str	The query string to run against all indices.
to_pandas	Optional[bool]	If True (default), returns a pandas DataFrame. If False, returns a list of dictionaries.

Execution flow:

1. Prints the query text with bold formatting.
2. Iterates over each index and its applicable retriever modes.
3. For each combination:
   • Records the start time.
   • Creates a TokenCountingHandler and CallbackManager.
   • Creates a query engine with the specified retriever mode (skips on ValueError).
   • Executes the query and prints the output with color coding.
   • Records duration, prompt tokens, completion tokens, and embedding tokens.
4. Prints the total number of combinations executed.
5. Returns results as a pandas DataFrame or list of dictionaries.

Result columns:

Column	Description
Index	The class name of the index.
Retriever Mode	The retriever mode string used.
Output	The string representation of the query response.
Duration	Wall-clock time in seconds.
Prompt Tokens	Number of prompt tokens consumed.
Completion Tokens	Number of completion tokens generated.
Embed Tokens	Number of embedding tokens used.

Dependencies

Module	Items Imported
time	Wall-clock timing of query execution.
llama_index.core.callbacks	CallbackManager, TokenCountingHandler for token metrics.
llama_index.core.indices.base	BaseIndex
llama_index.core.indices.list.base	ListRetrieverMode, SummaryIndex
llama_index.core.indices.tree.base	TreeIndex, TreeRetrieverMode
llama_index.core.indices.vector_store	VectorStoreIndex
llama_index.core.schema	Document
llama_index.core.utils	get_color_mapping, print_text for colored terminal output.
pandas (optional)	Used for DataFrame output. Raises ImportError if not installed and to_pandas=True.

Design Notes

• The Playground is designed for interactive experimentation and development workflows, not production use. It prints directly to stdout with ANSI formatting codes.
• Each index/mode combination creates its own TokenCountingHandler for isolated token tracking.
• If a particular retriever mode is incompatible with an index (raises ValueError), the combination is silently skipped via a try/except block.
• The color mapping assigns a unique terminal color to each index for visual differentiation in output.
• Pandas is an optional dependency: the method gracefully handles its absence by raising a descriptive ImportError.