Heuristic:Turboderp org Exllamav2 Dynamic Generator Tuning

Knowledge Sources	ExLlamaV2 Dynamic Generator Docs
Domains	Inference_Optimization, Concurrent_Batching
Last Updated	2026-02-15 00:00 GMT

Overview

Tuning guidance for ExLlamaV2DynamicGenerator: warm up before benchmarking, avoid speculative decoding with many parallel jobs, use multithreaded sampling only with 3+ active jobs, and understand the job skip starvation policy.

Description

The ExLlamaV2 dynamic generator is a complex concurrent batching system with many tunable behaviors. Several non-obvious defaults and thresholds affect real-world performance. This heuristic collects the tribal knowledge needed to get optimal performance from the generator.

Usage

Apply these tips when benchmarking ExLlamaV2 performance, tuning a production deployment, or debugging unexpected throughput issues with the dynamic generator.

The Insight (Rule of Thumb)

Action: Always call `generator.warmup()` before benchmarking or production use.
Value: The first inference run is always slow due to CUDA kernel autotuning. The warmup method generates 32 tokens with a dummy prompt and resets the page table.
Trade-off: ~1 second warmup cost. Without it, first-request latency will be significantly higher.

Action: Do not use speculative decoding with many parallel jobs.
Value: From the docstring: "speculative decoding with many parallel jobs is likely not advantageous."
Trade-off: Speculative decoding adds overhead per-token that is amortized by skipping forward passes. With many jobs, the batch already saturates GPU compute, making speculation wasteful.

Action: Multithreaded sampling only activates when active jobs >= `min_sampling_threads` (default 3).
Value: Below 3 active jobs, single-threaded sampling is used to avoid threading overhead.
Trade-off: For single-user scenarios, sampling is always single-threaded. Multi-threading helps when >= 3 jobs are running concurrently.

Action: Jobs can be skipped up to `max_skips` times (default 4) if they cannot fit in cache.
Value: Smaller jobs may start ahead of a large job that needs more cache pages. After 4 skips, the large job stalls the queue to prevent starvation.
Trade-off: Lower `max_skips` = fairer scheduling but potentially lower throughput. Higher = better throughput but large jobs wait longer.

Action: Vocabulary size is padded to multiples of 32 for CUDA kernel efficiency.
Value: `padded_vocab_size = ((vocab_size + 31) // 32) * 32`
Trade-off: Minimal memory overhead for significant kernel performance gain.

Action: LoRA adapters can only be changed when the job queue is completely empty.
Value: Assertion failure if `set_loras()` is called with jobs in the queue.
Trade-off: Plan LoRA switching during natural pauses in request processing.

Action: Classifier-Free Guidance (CFG) requires exactly batch_size=2 and is incompatible with filters.
Value: Positive and negative prompts are processed as a batch of 2. Constrained decoding filters only work with batch_size=1.
Trade-off: Cannot combine CFG with structured output constraints.

Action: During streaming, incomplete multibyte Unicode characters are held back until complete.
Value: If 1-4 replacement characters appear, the full token sequence is re-decoded. Above 5, output is held.
Trade-off: Slight latency increase for non-ASCII text, but prevents garbled output.

Reasoning

CUDA kernel autotuning selects optimal launch configurations (block sizes, shared memory) on first execution of each kernel variant. This happens once per unique problem size and persists for the process lifetime. Without warmup, the first real request bears this cost.

Speculative decoding trades compute (running a smaller draft model) for fewer synchronization points with the main model. With many parallel jobs, the GPU is already fully utilized by the main model's batched inference, so the draft model adds overhead without reducing total forward passes.

The job skip mechanism balances fairness (FIFO ordering) with throughput (small jobs can fill unused cache pages while waiting for a large job's cache pages to free up).

From `exllamav2/generator/dynamic.py:514-519`:

def warmup(self):
    """Warm up the generator by generating some text, making sure kernel autotune
    has time to complete."""
    self.generate("Once upon a time,", max_new_tokens = 32)
    self.reset_page_table()

From `exllamav2/generator/dynamic.py:1230-1238`:

if self.max_sampling_threads > 1 and \
   len(self.active_jobs) >= self.min_sampling_threads:
    mt_sample = True
else:
    mt_sample = False

From `exllamav2/generator/dynamic.py:530-531`:

assert not self.num_remaining_jobs(), \
    "LoRAs cannot be updated while there are jobs in the generator queue."

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment