Implementation:Vllm project Vllm LLM Init Speculative

Knowledge Sources	vLLM vLLM Docs
Domains	LLM Inference, Speculative Decoding, Engine Initialization
Last Updated	2026-02-08 13:00 GMT

Overview

Concrete tool for initializing the vLLM inference engine with speculative decoding enabled provided by vLLM.

Description

The LLM class constructor accepts a speculative_config keyword argument (passed through **kwargs to EngineArgs) that enables speculative decoding. When this parameter is provided, the engine loads both the target model and the draft mechanism (EAGLE head, draft model, or n-gram/MTP proposer) during initialization. The speculative_config dictionary is converted to a SpeculativeConfig dataclass internally via EngineArgs.create_speculative_config(), which enriches it with target_model_config and target_parallel_config before validation.

The LLM constructor remains identical in its explicit parameter signature whether or not speculation is used. The speculative_config is passed through **kwargs to EngineArgs, which stores it at EngineArgs.speculative_config (line 512 of arg_utils.py).

Usage

Use this API call to create a vLLM engine instance with speculative decoding. This is the primary entry point for offline speculative inference workloads.

Code Reference

Source Location

• Repository: vllm
• File: vllm/entrypoints/llm.py:L199-364 (LLM.__init__), vllm/engine/arg_utils.py:L512 (speculative_config field), vllm/engine/arg_utils.py:L1353-1378 (create_speculative_config)

Signature

class LLM:
    def __init__(
        self,
        model: str,
        *,
        runner: RunnerOption = "auto",
        convert: ConvertOption = "auto",
        tokenizer: str | None = None,
        tokenizer_mode: TokenizerMode | str = "auto",
        skip_tokenizer_init: bool = False,
        trust_remote_code: bool = False,
        allowed_local_media_path: str = "",
        allowed_media_domains: list[str] | None = None,
        tensor_parallel_size: int = 1,
        dtype: ModelDType = "auto",
        quantization: QuantizationMethods | None = None,
        revision: str | None = None,
        tokenizer_revision: str | None = None,
        seed: int = 0,
        gpu_memory_utilization: float = 0.9,
        swap_space: float = 4,
        cpu_offload_gb: float = 0,
        enforce_eager: bool = False,
        enable_return_routed_experts: bool = False,
        disable_custom_all_reduce: bool = False,
        hf_token: bool | str | None = None,
        hf_overrides: HfOverrides | None = None,
        mm_processor_kwargs: dict[str, Any] | None = None,
        pooler_config: PoolerConfig | None = None,
        compilation_config: int | dict[str, Any] | CompilationConfig | None = None,
        **kwargs: Any,  # speculative_config passed here
    ) -> None:
        ...

Import

from vllm import LLM

I/O Contract

Inputs

Name	Type	Required	Description
model	str	Yes	Path or Hugging Face Hub ID of the target model (e.g., "meta-llama/Llama-3.1-8B-Instruct").
speculative_config	dict[str, Any] or None	No	Dictionary describing the speculative decoding configuration. When None (default), speculative decoding is disabled. Passed via **kwargs.
trust_remote_code	bool	No	Whether to trust remote code from Hugging Face Hub. Defaults to False. Often needed for custom model architectures.
tensor_parallel_size	int	No	Number of GPUs for tensor parallelism. Defaults to 1.
gpu_memory_utilization	float	No	Fraction of GPU memory to use. Defaults to 0.9. May need to be increased when using speculative decoding with large draft models.
enforce_eager	bool	No	Disable CUDA graph compilation. Defaults to False. Sometimes needed for debugging speculative decoding.
dtype	str	No	Data type for model weights. Defaults to "auto".

Outputs

Name	Type	Description
llm	LLM	An initialized LLM engine instance with both target model and draft mechanism loaded, ready for speculative generation.

Usage Examples

Initialize with EAGLE Speculation

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    trust_remote_code=True,
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,
    speculative_config={
        "method": "eagle",
        "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B",
        "num_speculative_tokens": 3,
    },
)

Initialize with N-gram Speculation

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 3,
        "prompt_lookup_max": 5,
        "prompt_lookup_min": 2,
    },
)

Initialize with Draft Model Speculation

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    trust_remote_code=True,
    enforce_eager=True,
    speculative_config={
        "method": "draft_model",
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 3,
        "enforce_eager": True,
        "max_model_len": 16384,
    },
)

Initialize with MTP Speculation

from vllm import LLM

# Target model must natively support MTP heads
llm = LLM(
    model="deepseek-ai/DeepSeek-V3",
    trust_remote_code=True,
    tensor_parallel_size=8,
    speculative_config={
        "method": "mtp",
        "num_speculative_tokens": 2,
    },
)

Related Pages

Implements Principle

• Principle:Vllm_project_Vllm_Speculative_Engine_Initialization