Implementation:Mlc ai Mlc llm Generation Config

Overview

The Generation Config module defines the low-level generation configuration data class used internally by the MLC LLM engine. Located at python/mlc_llm/protocol/generation_config.py (33 lines), it is a Pydantic model that captures all parameters controlling text generation behavior, including sampling, penalties, stop criteria, and optional debug/grammar settings.

Purpose

This module provides a single GenerationConfig class that the MLC LLM engine uses internally to control inference-time generation parameters. It is distinct from external API request schemas and serves as the canonical internal representation of generation settings.

Key Components

GenerationConfig Class

class GenerationConfig(BaseModel):
    """The generation configuration dataclass.

    This is a config class used by Engine internally.
    """

    n: int = 1
    temperature: Optional[float] = None
    top_p: Optional[float] = None
    frequency_penalty: Optional[float] = None
    presence_penalty: Optional[float] = None
    repetition_penalty: Optional[float] = None
    logprobs: bool = False
    top_logprobs: int = 0
    logit_bias: Optional[Dict[int, float]] = None
    max_tokens: int = -1
    seed: Optional[int] = None
    stop_strs: Optional[List[str]] = None
    stop_token_ids: Optional[List[int]] = None
    response_format: Optional[RequestResponseFormat] = None
    debug_config: Optional[Optional[DebugConfig]] = None

Fields:

Field	Type	Default	Description
n	int	1	Number of completion sequences to generate
temperature	Optional[float]	None	Sampling temperature controlling randomness
top_p	Optional[float]	None	Nucleus sampling probability threshold
frequency_penalty	Optional[float]	None	Penalty applied based on token frequency in generated text
presence_penalty	Optional[float]	None	Penalty applied based on whether a token has appeared at all
repetition_penalty	Optional[float]	None	Penalty multiplier for repeated tokens
logprobs	bool	False	Whether to return log probabilities of output tokens
top_logprobs	int	0	Number of top log probabilities to return per token
logit_bias	Optional[Dict[int, float]]	None	Mapping from token IDs to bias values applied to logits before sampling
max_tokens	int	-1	Maximum number of tokens to generate; -1 represents infinite (no limit)
seed	Optional[int]	None	Random seed for deterministic generation
stop_strs	Optional[List[str]]	None	Strings that trigger generation to stop
stop_token_ids	Optional[List[int]]	None	Token IDs that trigger generation to stop
response_format	Optional[RequestResponseFormat]	None	Requested response format (e.g., JSON mode, grammar-guided generation)
debug_config	Optional[DebugConfig]	None	Optional debug configuration for engine-level debug options

Design Notes

• The max_tokens field uses -1 as a sentinel value to represent "infinite" or "no limit," rather than using None, which allows it to remain a plain int type.
• The response_format field references RequestResponseFormat from the OpenAI API protocol module, enabling grammar-guided or structured output generation.
• The debug_config field connects to the Debug Protocol module, providing access to debug features like ignore_eos and grammar execution modes.

Dependencies

• pydantic -- For BaseModel data validation
• mlc_llm.protocol.debug_protocol.DebugConfig -- Debug configuration class
• mlc_llm.protocol.openai_api_protocol.RequestResponseFormat -- Response format specification from the OpenAI-compatible API protocol

File Location

python/mlc_llm/protocol/generation_config.py