Implementation:Lm sys FastChat Category Label Pipeline

Knowledge Sources	Lm_sys_FastChat Chatbot Arena
Domains	Model_Evaluation, Data_Pipeline
Last Updated	2026-02-07 06:00 GMT

Overview

The Category Label Pipeline orchestrates batch category labeling of large conversation datasets by coordinating the CategoryClassifier with file I/O, checkpointing, and progress tracking.

Description

The label.py module implements a complete batch labeling pipeline for assigning topic categories to conversations collected from the Chatbot Arena. It reads raw conversation logs, extracts user prompts, invokes the CategoryClassifier for LLM-based labeling, and writes the enriched data back to disk with category annotations attached.

The pipeline is designed for robustness at scale. The process_batch function splits large datasets into manageable chunks and processes them sequentially, writing intermediate checkpoints so that labeling can be resumed if interrupted. Each batch is logged with timing information and error counts, enabling operators to monitor pipeline health during long-running jobs.

The label_conversations function serves as the primary API entry point, accepting a list of conversation records and returning them with category labels attached. The main function provides a command-line interface for running the pipeline end-to-end, reading input files, configuring the classifier, and writing labeled output. It supports configurable parameters for batch size, parallelism, output format, and model selection.

Usage

Use this module when you need to label a large corpus of arena conversations with topic categories. This is typically run as a batch job after collecting new battle logs and before recomputing category-specific leaderboard rankings. It is intended to be invoked from the command line or imported as a library in a larger data processing workflow.

Code Reference

Source Location

• Repository: Lm_sys_FastChat
• File: fastchat/serve/monitor/classify/label.py
• Lines: 1-439

Signature

def label_conversations(
    conversations: list[dict],
    classifier: CategoryClassifier,
    batch_size: int = 256,
    num_workers: int = 8,
) -> list[dict]:
    """Label a list of conversation records with topic categories."""
    ...

def process_batch(
    batch: list[dict],
    classifier: CategoryClassifier,
    num_workers: int = 8,
) -> list[dict]:
    """Process a single batch of conversations through the classifier."""
    ...

def main():
    """Command-line entry point for the category labeling pipeline."""
    ...

Import

from fastchat.serve.monitor.classify.label import label_conversations

I/O Contract

Inputs

Name	Type	Required	Description
conversations	list[dict]	Yes	List of conversation records, each containing user prompts and metadata
classifier	CategoryClassifier	Yes	An initialized CategoryClassifier instance for performing labeling
batch_size	int	No	Number of conversations to process per batch (default: 256)
num_workers	int	No	Number of parallel API workers (default: 8)
input_file	str	Yes (CLI)	Path to input JSON/JSONL file containing conversation logs
output_file	str	Yes (CLI)	Path to output file for labeled conversations

Outputs

Name	Type	Description
labeled_conversations	list[dict]	Conversation records enriched with a "category" field containing assigned labels
output_file	JSON/JSONL file	Written to disk with category annotations for each conversation

Usage Examples

from fastchat.serve.monitor.classify.category import CategoryClassifier
from fastchat.serve.monitor.classify.label import label_conversations

# Initialize classifier
categories = ["coding", "math", "reasoning", "creative_writing", "general"]
classifier = CategoryClassifier(category_list=categories, model_name="gpt-4")

# Load conversation records
import json
with open("arena_battles_2024_01.json") as f:
    conversations = json.load(f)

# Label all conversations
labeled = label_conversations(
    conversations,
    classifier=classifier,
    batch_size=128,
    num_workers=4,
)

# Inspect results
for conv in labeled[:5]:
    print(f"Category: {conv['category']}, Prompt: {conv['conversation'][0]['content'][:60]}...")

Command-Line Usage

python -m fastchat.serve.monitor.classify.label \
    --input arena_battles.json \
    --output arena_battles_labeled.json \
    --model gpt-4 \
    --batch-size 256 \
    --num-workers 8

Related Pages

• Principle:Lm_sys_FastChat_LLM_Prompt_Classification
• Implements: Principle:Lm_sys_FastChat_LLM_Prompt_Classification
• Environment:Lm_sys_FastChat_GPU_CUDA_Inference
• Lm_sys_FastChat_Category_Classifier - Core classifier used by this pipeline
• Lm_sys_FastChat_Clean_Battle_Data - Produces cleaned data fed into this pipeline
• Lm_sys_FastChat_Elo_Analysis - Consumes labeled data for category-specific analysis