Implementation:Ggml org Ggml Gpt sample top k top p

Template:Implementation

Summary

gpt_sample_top_k_top_p is a C++ function that samples a single token from the model's output logits using combined top-k and top-p (nucleus) sampling with temperature scaling.

API

gpt_vocab::id gpt_sample_top_k_top_p(
    const gpt_vocab & vocab,
    const float * logits,
    int top_k,
    double top_p,
    double temp,
    std::mt19937 & rng
)

Source

• File: examples/common.cpp, lines 401-479
• Repository: https://github.com/ggml-org/ggml

Parameters

Parameter	Type	Description
vocab	const gpt_vocab &	Vocabulary object; used to determine n_vocab (vocabulary size)
logits	const float *	Raw model output logits, array of size n_vocab
top_k	int	Number of highest-probability tokens to retain for top-k filtering; 0 disables top-k
top_p	double	Cumulative probability threshold for nucleus sampling; 1.0 disables top-p
temp	double	Temperature for softmax scaling; 1.0 is neutral (no scaling)
rng	std::mt19937 &	Mersenne Twister random number generator for stochastic sampling

Returns

• Type: gpt_vocab::id
• Description: The sampled token ID from the filtered and renormalized distribution

Implementation

The function applies the following pipeline in order:

1. Temperature scaling — Divide all logits by temp
2. Sort logits — Sort token-logit pairs in descending order by logit value
3. Top-k filter — If top_k > 0, truncate the sorted list to the first top_k entries
4. Softmax — Convert the filtered logits into a probability distribution
5. Top-p filter — If top_p < 1.0, walk the sorted probabilities and keep tokens until the cumulative probability reaches top_p, then truncate
6. Sample — Draw a token from the remaining candidates using std::discrete_distribution

Dependencies

Standard C++ headers only:

• <algorithm> — for sorting logits
• <random> — for std::mt19937 and std::discrete_distribution
• <numeric> — for accumulation during softmax and cumulative probability computation

Related

• Principle:Ggml_org_Ggml_Token_Sampling
• Environment:Ggml_org_Ggml_C_Cpp_Build_Environment
• Heuristic:Ggml_org_Ggml_Sampling_Parameter_Defaults