Principle:InternLM Lmdeploy Generation Configuration
| Knowledge Sources | |
|---|---|
| Domains | Text_Generation, Sampling |
| Last Updated | 2026-02-07 15:00 GMT |
Overview
A parameterization of the text generation sampling strategy controlling output length, randomness, repetition, and stopping conditions.
Description
Generation Configuration encapsulates all parameters that control how tokens are sampled from the model's output probability distribution. These parameters include:
- Temperature: Controls randomness (0 = deterministic, higher = more random)
- Top-p (nucleus sampling): Cumulative probability threshold for candidate tokens
- Top-k: Maximum number of candidate tokens to consider
- Repetition penalty: Penalizes tokens that have already appeared
- Max new tokens: Hard limit on output length
- Stop words: Token sequences that trigger generation termination
- Min new tokens: Minimum tokens before stop words are checked
The configuration supports advanced features like n-best generation (multiple completions per prompt), logprobs output, and guided decoding constraints (JSON schema, regex).
Usage
Use this whenever you need to control the quality and characteristics of generated text. Set temperature=0 for deterministic factual outputs, higher temperatures for creative text. Adjust top_p and top_k together for fine-grained diversity control. Use stop_words for structured output parsing.
Theoretical Basis
Token sampling applies transformations to the raw logits:
Then filtering:
- Top-k filter: Keep only top k logits, set rest to -inf
- Top-p filter: Sort by probability, keep smallest set summing to p
Finally: Failed to parse (syntax error): {\displaystyle P(x) = \text{softmax}(\text{filtered\_logits}')}