Implementation:Hiyouga LLaMA Factory API App

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	API, Inference Serving
Last Updated	2026-02-06 19:00 GMT

Overview

API App is a FastAPI application factory that creates an OpenAI-compatible HTTP API server for LLaMA Factory inference engines.

Description

The module provides the create_app factory function which constructs a FastAPI application with CORS middleware, optional Bearer token authentication via the API_KEY environment variable, and three OpenAI-compatible REST endpoints. A background sweeper task periodically runs torch_gc to reclaim GPU memory when the HuggingFace engine is active. The run_api convenience function instantiates a ChatModel, builds the app, and launches a uvicorn server on a configurable host and port.

Usage

Use this module when you need to expose a LLaMA Factory model as an OpenAI-compatible HTTP API server. Call run_api() directly to start the server, or use create_app(chat_model) to integrate the FastAPI app into a custom deployment setup.

Code Reference

Source Location

Repository: Hiyouga_LLaMA_Factory
File: src/llamafactory/api/app.py
Lines: 1-133

Signature

def create_app(chat_model: "ChatModel") -> "FastAPI":
    ...

def run_api() -> None:
    ...

async def sweeper() -> None:
    ...

@asynccontextmanager
async def lifespan(app: "FastAPI", chat_model: "ChatModel"):
    ...

Import

from llamafactory.api.app import create_app, run_api

I/O Contract

Inputs

Name	Type	Required	Description
chat_model	ChatModel	Yes	The initialized chat model instance used by all endpoints
API_KEY	str (env var)	No	Bearer token for API authentication; if unset, no auth is required
API_HOST	str (env var)	No	Host address for the server (default: "0.0.0.0")
API_PORT	str (env var)	No	Port number for the server (default: "8000")
FASTAPI_ROOT_PATH	str (env var)	No	Root path prefix for the FastAPI application
API_MODEL_NAME	str (env var)	No	Model name returned by /v1/models (default: "gpt-3.5-turbo")

Outputs

Name	Type	Description
FastAPI app	FastAPI	Configured FastAPI application with OpenAI-compatible endpoints
GET /v1/models	ModelList	Returns the list of available models
POST /v1/chat/completions	ChatCompletionResponse or EventSourceResponse	Chat completion (streaming or non-streaming)
POST /v1/score/evaluation	ScoreEvaluationResponse	Reward model score evaluation

Usage Examples

# Start the API server directly
from llamafactory.api.app import run_api
run_api()

# Or create the app for custom deployment
from llamafactory.chat import ChatModel
from llamafactory.api.app import create_app

chat_model = ChatModel({"model_name_or_path": "meta-llama/Llama-2-7b-chat-hf"})
app = create_app(chat_model)
# Use with uvicorn or any ASGI server

Related Pages

Hiyouga_LLaMA_Factory_API_Chat - Chat completion response generation functions
Hiyouga_LLaMA_Factory_API_Protocol - Pydantic data models for API request/response schemas
Hiyouga_LLaMA_Factory_Chat_Model - Unified ChatModel facade used by the API
Hiyouga_LLaMA_Factory_Base_Engine - Abstract base class for inference engines

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment