Implementation:Turboderp org Exllamav2 Sampling H

Knowledge Sources	Turboderp_org_Exllamav2
Domains	Sampling, C_Extension
Last Updated	2026-02-15 00:00 GMT

Overview

C++ header file declaring the function signatures for 15+ CPU-side sampling algorithms used in the ExLlamaV2 token generation pipeline.

Description

sampling.h is the central header that defines the complete suite of sampling operations available in ExLlamaV2's C++ extension layer. Each function operates on raw float arrays representing logit or probability distributions over the vocabulary.

The declared functions include:

• softmax_cpu -- Converts raw logits to a probability distribution with temperature scaling and optional exponent, returning the index of the maximum logit.
• top_k_cpu -- Truncates the distribution to the top-K most probable candidates.
• top_p_cpu -- Applies nucleus sampling by keeping candidates whose cumulative probability exceeds the threshold.
• top_a_cpu -- Filters candidates based on an absolute threshold relative to the top probability.
• min_p_cpu -- Removes candidates below a minimum probability fraction of the most likely token.
• tfs_cpu -- Tail-free sampling using the second derivative of the sorted probability distribution.
• typical_cpu -- Typical sampling based on information-theoretic surprise values.
• mirostat_pre_cpu / mirostat_post_cpu -- Two-phase Mirostat adaptive sampling: pre-filtering and post-update of the target surprise parameter mu.
• xtc_cpu -- Exclude Top Choices sampling with a probability-based mask and threshold.
• multinomial_cpu -- Weighted random selection from the filtered candidate distribution.
• apply_rep_penalty_cpu -- Applies repetition penalty with frequency and presence alpha parameters, plus sustain and decay windowing over a token sequence.
• post_softmax_temperature -- Dynamic temperature adjustment applied after softmax, with min/max bounds and a temperature exponent.
• normalize_cpu -- Renormalizes a truncated probability array to sum to 1.0.
• sort_descending / pre_sort_descending -- Sorts candidates in descending probability order for subsequent filtering stages.

The header also declares profiling utilities (profile_results, profile_start, profile_stop) used to instrument sampling performance.

Usage

This header is included by the various sampling implementation files (e.g., sampling.cpp, sampling_avx2.cpp) and by the Python extension binding layer. It provides the contract for all CPU-side sampling operations that are called from the Python ExLlamaV2Sampler during token generation.

Code Reference

Source Location

• Repository: Turboderp_org_Exllamav2
• File: exllamav2/exllamav2_ext/cpp/sampling.h
• Lines: 1-156

Signature

int softmax_cpu(
    const int vocab_size,
    const float temperature,
    const float* logits,
    const bool* logits_filter,
    const float exponent,
    float* output
);

int top_k_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    int top_k,
    int maxlogit = -1
);

int top_p_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float top_p
);

int top_a_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float top_a
);

int min_p_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float min_p
);

int tfs_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float tfs
);

int typical_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float typical
);

int mirostat_pre_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float mirostat_mu,
    float mirostat_tau,
    float mirostat_eta
);

float mirostat_post_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float mirostat_mu,
    float mirostat_tau,
    float mirostat_eta
);

int xtc_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    bool* xtc_mask,
    float xtc_probability,
    float xtc_threshold
);

int multinomial_cpu(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float random
);

void apply_rep_penalty_cpu(
    const int vocab_size,
    const uint64_t* sequence,
    const float penalty_max,
    const int sustain,
    const int decay,
    const float alpha_frequency,
    const float alpha_presence,
    const int seq_len,
    float* logits
);

int post_softmax_temperature(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    float temp,
    float min_temp,
    float max_temp,
    float temp_exponent
);

void normalize_cpu(
    const int num_candidates,
    float* probs
);

int sort_descending(
    const int num_candidates,
    float* temp_probs,
    int* temp_indices,
    int max_index
);

Import

#include "sampling.h"

I/O Contract

Function	Input	Output	Return
softmax_cpu	vocab_size, temperature, logits array, optional logits_filter, exponent	Probability distribution written to output array	Index of max logit (int)
top_k_cpu	num_candidates, sorted probs + indices, top_k value	Truncated arrays in-place	New candidate count (int)
top_p_cpu	num_candidates, sorted probs + indices, top_p threshold	Truncated arrays in-place	New candidate count (int)
top_a_cpu	num_candidates, sorted probs + indices, top_a threshold	Truncated arrays in-place	New candidate count (int)
min_p_cpu	num_candidates, sorted probs + indices, min_p fraction	Truncated arrays in-place	New candidate count (int)
tfs_cpu	num_candidates, sorted probs + indices, tfs threshold	Truncated arrays in-place	New candidate count (int)
typical_cpu	num_candidates, sorted probs + indices, typical threshold	Truncated arrays in-place	New candidate count (int)
mirostat_pre_cpu	num_candidates, probs + indices, mu, tau, eta	Truncated arrays in-place	New candidate count (int)
mirostat_post_cpu	num_candidates, probs + indices, mu, tau, eta	Updated mu value	New mu (float)
xtc_cpu	num_candidates, probs + indices, xtc_mask, probability, threshold	Mask written in-place	New candidate count (int)
multinomial_cpu	num_candidates, probs + indices, random float [0,1)	Selected token	Selected index (int)
apply_rep_penalty_cpu	vocab_size, sequence, penalty params	Logits modified in-place	void
post_softmax_temperature	num_candidates, probs + indices, temp params	Probs modified in-place	New candidate count (int)
normalize_cpu	num_candidates, probs array	Probs normalized in-place to sum=1.0	void
sort_descending	num_candidates, probs + indices, max_index	Arrays sorted in descending order in-place	New candidate count (int)

Usage Examples

// Typical sampling pipeline in C++ extension
float output[vocab_size];
int max_idx = softmax_cpu(vocab_size, temperature, logits, logits_filter, 1.0f, output);

float temp_probs[vocab_size];
int temp_indices[vocab_size];
memcpy(temp_probs, output, vocab_size * sizeof(float));
for (int i = 0; i < vocab_size; i++) temp_indices[i] = i;

int n = pre_sort_descending(vocab_size, temp_probs, temp_indices);
n = sort_descending(n, temp_probs, temp_indices, max_idx);
n = top_k_cpu(n, temp_probs, temp_indices, 50);
n = top_p_cpu(n, temp_probs, temp_indices, 0.9f);
normalize_cpu(n, temp_probs);
int selected = multinomial_cpu(n, temp_probs, temp_indices, random_float);

Related Pages

• Turboderp_org_Exllamav2_Softmax_AVX2 -- AVX2-optimized implementation of softmax_cpu
• Turboderp_org_Exllamav2_Ext_Norm -- GPU normalization operations