Implementation:InternLM Lmdeploy StopCriteriaKernels

Knowledge Sources	InternLM_Lmdeploy
Domains	GPU_Kernels, Sampling
Last Updated	2026-02-07 15:00 GMT

Overview

CUDA kernels for checking stop criteria during autoregressive generation, including stop-word matching and maximum length enforcement.

Description

This header declares two stopping-criteria kernels used during the decoding loop. invokeStopWordsCriterion_v2() checks whether any sequence in the batch has generated a stop-word sequence by comparing the tail of each sequence's generated token IDs against a list of stop words. When a match is found, the corresponding entry in the finished array is set to true. invokeLengthCriterion_v2() compares each sequence's current length against its per-sequence length limit and marks sequences as finished when they reach or exceed their maximum allowed length.

Usage

Use these kernels at each decoding step to determine which sequences should stop generating. They are invoked after token sampling and before the next iteration of the generation loop.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/kernels/stop_criteria_kernels.h

Signature

void invokeStopWordsCriterion_v2(
    const int**  token_ids_ptrs,
    const int*   sequence_length,
    const int*   stop_words,
    bool*        finished,
    int          stop_words_len,
    int          batch_size,
    cudaStream_t stream);

void invokeLengthCriterion_v2(
    bool*        finished,
    const int*   sequence_length,
    const int*   sequence_length_limit,
    int          batch_size,
    cudaStream_t stream);

Import

#include "src/turbomind/kernels/stop_criteria_kernels.h"

I/O Contract

Inputs

Name	Type	Required	Description
token_ids_ptrs	const int**	Yes	Array of pointers to each sequence's generated token IDs
sequence_length	const int*	Yes	Current length of each sequence
stop_words	const int*	Yes	Flattened stop-word sequences to match against
stop_words_len	int	Yes	Length of the stop_words buffer
sequence_length_limit	const int*	Yes	Per-sequence maximum length limit
batch_size	int	Yes	Number of sequences in the batch

Outputs

Name	Type	Description
finished	bool*	Per-sequence flag set to true when stop criterion is met

Usage Examples

using namespace turbomind;

// Check stop words
invokeStopWordsCriterion_v2(
    token_ptrs, seq_lengths, stop_words, finished,
    stop_words_len, batch_size, stream);

// Check length limit
invokeLengthCriterion_v2(
    finished, seq_lengths, max_lengths, batch_size, stream);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime