Implementation:Turboderp org Exllamav2 WebSocket Actions

Knowledge Sources	Turboderp_org_Exllamav2
Domains	Server, WebSocket, Text_Generation
Last Updated	2026-02-15 00:00 GMT

Overview

The WebSocket_Actions module defines the request handler functions for the ExLlamaV2 WebSocket server, including token estimation, text trimming, streaming inference, and interrupt control.

Description

This module contains the action handler functions that are dispatched by ExLlamaV2WebSocketServer. The dispatch() function routes incoming JSON requests based on the action field to the appropriate handler. Each handler receives the parsed request dict, the WebSocket connection, the server instance, and a pre-initialized response dict.

The available actions are:

• echo - Returns the response with only the echoed request/response IDs. Used as a ping/health check.

• estimate_token - Encodes the provided text using the server's tokenizer and returns the token count as num_tokens. Useful for clients to estimate prompt length before sending an inference request.

• lefttrim_token - Encodes the input text, trims from the left to keep only trimmed_length tokens from the right, then decodes back to text. Returns trimmed_text. This is used for context window management.

• infer - The main generation handler (async). Acquires the model_lock to ensure exclusive model access. Configures sampler settings from request parameters (top_k, top_p, top_a, min_p, typical, temperature, skew, repetition/frequency/presence penalties). Handles prompt tokenization, context overflow trimming, custom BOS tokens, stop conditions, and token healing. Generates tokens in a streaming loop, sending response_type: "chunk" messages for each generated piece of text. Supports stream_full mode where each chunk message includes the full response so far. Generation terminates on EOS, max tokens reached, or the stop_signal being set. The final response has response_type: "full" with the complete text and stop_reason ("eos", "num_tokens", or "interrupted").

• stop - Sets the server's stop_signal event to interrupt any active inference. The next iteration of the infer loop will detect the signal and terminate.

Usage

These functions are not called directly by users. They are invoked by ExLlamaV2WebSocketServer.main() via the dispatch() function when a JSON message arrives on a WebSocket connection. Clients interact with these actions by sending appropriately formatted JSON messages.

Code Reference

Source Location

• Repository: Turboderp_org_Exllamav2
• File: exllamav2/server/websocket_actions.py
• Lines: 1-259

Signature

async def dispatch(request: dict, ws, server) -> None: ...

def echo(request: dict, ws, server, response: dict) -> None: ...

def estimate_token(request: dict, ws, server, response: dict) -> None: ...

def lefttrim_token(request: dict, ws, server, response: dict) -> None: ...

async def infer(request: dict, ws, server, response: dict) -> None: ...

def stop(request: dict, ws, server, response: dict) -> None: ...

Import

from exllamav2.server import websocket_actions

I/O Contract

dispatch()

Parameter	Type	Description
request	dict	Parsed JSON request with an "action" field
ws	WebSocketServerProtocol	WebSocket connection for sending responses
server	ExLlamaV2WebSocketServer	Server instance providing model, tokenizer, generator, and locks

infer() Request Fields

Field	Type	Required	Description
action	str	Yes	Must be "infer"
text	str	Yes	Input prompt text
max_new_tokens	int	Yes	Maximum number of tokens to generate
stream	bool	Yes	Whether to stream chunk responses
stream_full	bool	No	If True, each chunk includes full response so far
top_k	int	No	Top-K sampling (default: 100, 0 to disable)
top_p	float	No	Top-P / nucleus sampling (default: 0.8, 0 to disable)
top_a	float	No	Top-A sampling threshold (default: 0)
min_p	float	No	Min-P sampling threshold (default: 0)
typical	float	No	Typical sampling threshold (default: 0)
temperature	float	No	Sampling temperature (default: 0.9)
skew	float	No	Skew factor (default: 0.0)
rep_pen	float	No	Repetition penalty (default: 1.05)
freq_pen	float	No	Frequency penalty (default: 0.0)
pres_pen	float	No	Presence penalty (default: 0.0)
customBos	str	No	Custom BOS token prepended to prompt
stop_conditions	int]	No	Additional stop strings/token IDs
token_healing	bool	No	Enable token healing (default: False)
tag	str	No	Echoed in response for client-side correlation

infer() Response Format

Field	Type	Description
action	str	"infer"
response_type	str	"chunk" for streaming, "full" for final
chunk	str	Next text chunk (streaming only)
response	str	Full generated text (final response, or partial when stream_full=True)
util_text	str	Input context after overflow trimming (final response only)
stop_reason	str	"eos", "num_tokens", or "interrupted" (final response only)
tag	str	Echoed tag (if provided in request)
request_id	str	Echoed request ID (if provided)
response_id	str	Echoed response ID (if provided)

Usage Examples

# Client-side Python example using the websockets library
import asyncio
import json
import websockets

async def chat():
    async with websockets.connect("ws://localhost:7862") as ws:
        # Send an inference request
        request = {
            "action": "infer",
            "text": "Explain quantum computing in simple terms:",
            "max_new_tokens": 200,
            "stream": True,
            "temperature": 0.7,
            "top_p": 0.9,
            "top_k": 50,
            "rep_pen": 1.05,
            "stop_conditions": ["\n\n"],
            "token_healing": True,
        }
        await ws.send(json.dumps(request))

        # Receive streaming chunks
        while True:
            msg = json.loads(await ws.recv())
            if msg["response_type"] == "chunk":
                print(msg.get("chunk", ""), end="", flush=True)
            elif msg["response_type"] == "full":
                print(f"\n[Stop reason: {msg['stop_reason']}]")
                break

asyncio.run(chat())

# Estimate token count
# {"action": "estimate_token", "text": "Hello world"}
# Response: {"action": "estimate_token", "num_tokens": 2}

# Left-trim to fit context
# {"action": "lefttrim_token", "text": "...", "trimmed_length": 2048}
# Response: {"action": "lefttrim_token", "trimmed_text": "..."}

# Interrupt active generation
# {"action": "stop"}

Related Pages

• Turboderp_org_Exllamav2_ExLlamaV2WebSocketServer - The WebSocket server that dispatches to these action handlers
• Turboderp_org_Exllamav2_ExLlamaV2DynamicGeneratorAsync - Alternative async generation interface for more advanced use cases
• Turboderp_org_Exllamav2_ExLlamaV2TokenizerBase - Base tokenizer class used for encoding/decoding in these actions