Implementation:Predibase Lorax Chat Completion Request

Knowledge Sources	LoRAX
Domains	API_Design, Adapter_Management
Last Updated	2026-02-08 02:00 GMT

Overview

Concrete tool for parsing OpenAI-format chat requests and extracting adapter routing information provided by the ChatCompletionRequest Rust struct.

Description

The ChatCompletionRequest struct in router/src/lib.rs deserializes the OpenAI-format JSON request body. Its try_into_generate() method converts the request into internal GenerateParameters, mapping the model field to adapter_id and extracting generation parameters (temperature, top_p, max_tokens, etc.).

Usage

Used internally by the /v1/chat/completions handler in the router. Not called directly by users.

Code Reference

Source Location

• Repository: LoRAX
• File: router/src/lib.rs
• Lines: 667-857

Signature

#[derive(Clone, Deserialize, ToSchema)]
pub(crate) struct ChatCompletionRequest {
    pub model: String,                    // Used as adapter_id
    pub messages: Vec<Message>,           // Conversation history
    pub max_tokens: Option<i32>,          // Max generation length
    pub temperature: Option<f32>,         // Sampling temperature
    pub top_p: Option<f32>,               // Nucleus sampling
    pub stream: Option<bool>,             // SSE streaming
    pub stop: Option<Vec<String>>,        // Stop sequences
    pub seed: Option<u64>,                // Random seed
    pub response_format: Option<ResponseFormat>,  // JSON schema
    pub tools: Option<Vec<Tool>>,         // Function calling
    pub tool_choice: Option<ToolChoice>,  // Tool selection
    pub adapter_id: Option<String>,       // Explicit adapter override
    pub adapter_source: Option<String>,   // Adapter source type
    pub api_token: Option<String>,        // Auth for private adapters
}

impl ChatCompletionRequest {
    pub fn try_into_generate(
        self,
        infer: &Infer,
    ) -> Result<(GenerateParameters, Vec<Message>, ...), ...>;
}

Import

// Internal module
use crate::ChatCompletionRequest;

I/O Contract

Inputs

Name	Type	Required	Description
model	String	Yes	Adapter ID or base model ID
messages	Vec[Message]	Yes	Conversation history
max_tokens	Option[i32]	No	Max tokens to generate
temperature	Option[f32]	No	Sampling temperature
stream	Option[bool]	No	Enable SSE streaming

Outputs

Name	Type	Description
GenerateParameters	struct	Internal parameters with adapter_id extracted
messages	Vec[Message]	Validated message array for template rendering

Usage Examples

Client-Side Request

from openai import OpenAI

client = OpenAI(base_url="http://localhost:3000/v1", api_key="x")

# model = adapter ID for routing
response = client.chat.completions.create(
    model="arnavgrg/codealpaca-qlora",  # LoRA adapter
    messages=[
        {"role": "user", "content": "Write a Python function to sort a list"}
    ],
    temperature=0.7,
    max_tokens=200,
)

Related Pages

Implements Principle

• Principle:Predibase_Lorax_Chat_Adapter_Routing