Implementation:Hiyouga LLaMA Factory API Chat

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	API, Multimodal Processing
Last Updated	2026-02-06 19:00 GMT

Overview

API Chat is the central translation layer between the OpenAI API protocol and the internal ChatModel interface, handling request validation, multimodal processing, and response formatting.

Description

The module provides three main async functions for generating API responses. _process_request validates and transforms OpenAI-format chat messages into the internal format, handling role mapping (user/assistant/system/function/tool), multimodal inputs (images, videos, and audio via base64, URL, or local file paths with LFI/SSRF security checks), and tool call extraction. create_chat_completion_response produces a complete non-streaming response, while create_stream_chat_completion_response is an async generator that yields SSE-formatted chunks. create_score_evaluation_response handles reward model scoring.

Usage

Use this module indirectly through the API endpoints defined in app.py. The functions are called by the FastAPI route handlers to process incoming chat completion and score evaluation requests.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/api/chat.py
• Lines: 1-291

Signature

def _process_request(
    request: "ChatCompletionRequest",
) -> tuple[
    list[dict[str, str]],
    Optional[str],
    Optional[str],
    Optional[list["ImageInput"]],
    Optional[list["VideoInput"]],
    Optional[list["AudioInput"]],
]:
    ...

async def create_chat_completion_response(
    request: "ChatCompletionRequest", chat_model: "ChatModel"
) -> "ChatCompletionResponse":
    ...

async def create_stream_chat_completion_response(
    request: "ChatCompletionRequest", chat_model: "ChatModel"
) -> AsyncGenerator[str, None]:
    ...

async def create_score_evaluation_response(
    request: "ScoreEvaluationRequest", chat_model: "ChatModel"
) -> "ScoreEvaluationResponse":
    ...

Import

from llamafactory.api.chat import (
    create_chat_completion_response,
    create_stream_chat_completion_response,
    create_score_evaluation_response,
)

I/O Contract

Inputs

Name	Type	Required	Description
request	ChatCompletionRequest	Yes	OpenAI-format chat completion request with messages, model, tools, and generation parameters
chat_model	ChatModel	Yes	The initialized chat model instance for inference
request (score)	ScoreEvaluationRequest	Yes (for scoring)	Request containing messages to score and optional max_length

Outputs

Name	Type	Description
ChatCompletionResponse	ChatCompletionResponse	Complete response with choices, usage stats, and finish reason
AsyncGenerator[str, None]	SSE stream	Yields JSON-serialized stream chunks followed by "[DONE]"
ScoreEvaluationResponse	ScoreEvaluationResponse	Reward model scores for the input messages

Usage Examples

# Non-streaming chat completion (called internally by app.py)
response = await create_chat_completion_response(request, chat_model)

# Streaming chat completion
async for chunk in create_stream_chat_completion_response(request, chat_model):
    # chunk is a JSON string or "[DONE]"
    pass

# Score evaluation
score_response = await create_score_evaluation_response(score_request, chat_model)

Related Pages

• Hiyouga_LLaMA_Factory_API_App - FastAPI application factory that routes to these functions
• Hiyouga_LLaMA_Factory_API_Protocol - Pydantic models for request/response schemas
• Hiyouga_LLaMA_Factory_Chat_Model - ChatModel facade providing achat and astream_chat methods