Implementation:Run llama Llama index FinetuningHandler Event Capture

Overview

The FinetuningHandler Event Capture implementation provides the callback-based mechanism for collecting LLM interaction data during normal pipeline operation. The OpenAIFineTuningHandler class extends BaseFinetuningHandler to intercept all LLM events, accumulate message histories, and serialize them into OpenAI-compatible JSONL format for finetuning.

Source File

File: llama-index-finetuning/llama_index/finetuning/callbacks/finetuning_handler.py
Lines: 9-160
Import: from llama_index.finetuning.callbacks import OpenAIFineTuningHandler

Class Hierarchy

Class	Parent	Description
`BaseFinetuningHandler`	`BaseCallbackHandler`	Abstract base providing event capture logic for all finetuning handlers
`OpenAIFineTuningHandler`	`BaseFinetuningHandler`	Concrete implementation that serializes to OpenAI JSONL format

BaseFinetuningHandler

Constructor

class BaseFinetuningHandler(BaseCallbackHandler):
    def __init__(self) -> None:

Parameters: None

Internal State:

_finetuning_events: Dict[str, List[Any]] -- Maps event IDs to lists of ChatMessage objects (input messages + response)
_function_calls: Dict[str, List[Any]] -- Maps event IDs to function/tool call definitions when present

on_event_start

def on_event_start(
    self,
    event_type: CBEventType,
    payload: Optional[Dict[str, Any]] = None,
    event_id: str = "",
    parent_id: str = "",
    **kwargs: Any,
) -> str:

Parameters:

Parameter	Type	Description
`event_type`	`CBEventType`	The type of callback event (only `CBEventType.LLM` is processed)
`payload`	`Optional[Dict[str, Any]]`	Event payload containing messages and additional kwargs
`event_id`	`str`	Unique identifier for this event
`parent_id`	`str`	ID of the parent event in the trace hierarchy

Returns: str -- The event ID

Behavior:

Only processes events where event_type == CBEventType.LLM
Extracts messages from payload via EventPayload.PROMPT (raw string prompts, wrapped as user messages) or EventPayload.MESSAGES (structured chat messages)
Stores extracted messages in _finetuning_events[event_id]
If EventPayload.ADDITIONAL_KWARGS contains a "functions" key, stores function definitions in _function_calls[event_id]

on_event_end

def on_event_end(
    self,
    event_type: CBEventType,
    payload: Optional[Dict[str, Any]] = None,
    event_id: str = "",
    **kwargs: Any,
) -> None:

Parameters:

Parameter	Type	Description
`event_type`	`CBEventType`	The type of callback event
`payload`	`Optional[Dict[str, Any]]`	Event payload containing the LLM response
`event_id`	`str`	Unique identifier matching the start event

Behavior:

Only processes LLM events that have a corresponding start event in _finetuning_events
Extracts the response: if the response is a raw string, wraps it as an assistant ChatMessage; otherwise extracts the .message attribute from the response object
Appends the response message to _finetuning_events[event_id], completing the conversation

OpenAIFineTuningHandler

get_finetuning_events

def get_finetuning_events(self) -> Dict[str, Dict[str, Any]]:

Returns: Dict[str, Dict[str, Any]] -- A dictionary keyed by event ID, where each value has:

"messages": All messages except the last (the input messages)
"response": The last message (the assistant response)

save_finetuning_events

def save_finetuning_events(self, path: str) -> None:

Parameters:

Parameter	Type	Description
`path`	`str`	File path to write the JSONL output

Behavior:

Calls get_finetuning_events() to separate messages from responses
Converts all ChatMessage objects to OpenAI message dictionaries using to_openai_message_dicts(all_messages, drop_none=True)
Constructs a JSON object with a "messages" key for each event
If function calls exist for an event, adds a "functions" key to the JSON object
Writes each JSON object as a single line to the output file (JSONL format)
Prints the number of examples written

Usage Example

from llama_index.finetuning.callbacks import OpenAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager
from llama_index.core import Settings

# Create handler and register it
finetuning_handler = OpenAIFineTuningHandler()
callback_manager = CallbackManager([finetuning_handler])
Settings.callback_manager = callback_manager

# After running queries through a query engine...
# Access raw events programmatically
events = finetuning_handler.get_finetuning_events()
print(f"Collected {len(events)} training examples")

# Save to JSONL for finetuning
finetuning_handler.save_finetuning_events("finetuning_events.jsonl")
# Output: Wrote N examples to finetuning_events.jsonl

Output Format

Each line in the output JSONL file follows this structure:

# Without function calls
{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

# With function calls
{"messages": [{"role": "user", "content": "..."}], "functions": [{"name": "...", "parameters": {...}}]}

Knowledge Sources

LlamaIndex Finetuning Callbacks Source

Metadata

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment