Implementation:InternLM Lmdeploy LogitsProcessor

Knowledge Sources	InternLM_Lmdeploy
Domains	Text Generation, Logits Manipulation
Last Updated	2026-02-07 15:00 GMT

Overview

Applies logits transformations including repetition penalty, bad word banning, minimum length penalty, and temperature scaling before token sampling.

Description

The LogitsProcessor class implements the logits modification pipeline that runs between the model's forward pass and token sampling. It inherits from BaseGenerationParam and maintains per-phase Data structs on device memory plus a shared host buffer for staging.

The processing pipeline follows the same order as the Hugging Face Transformers library:

1. Repetition penalty: Applies ApplyRepetitionPenalty to penalize already-generated tokens. Skipped if all requests in the batch have a penalty of 1.0.

2. Bad words banning: Uses BanBadWords to set logits of forbidden token sequences to negative infinity. Bad word lists are initialized from each request's GenerationConfig::bad_ids.

3. Minimum length penalty: Enforces minimum generation length by penalizing end-of-sequence tokens via invokeMinLengthPenalty when the current sequence length is below the minimum. Uses per-request end-of-sequence ID lists (up to kMaxEndIdsSize).

4. Temperature scaling: Divides logits by the temperature value using invokeBatchApplyTemperaturePenalty_v2. Skipped if all temperatures are 1.0.

Setup(): Iterates over the batch's request caches to extract per-request parameters, sets boolean flags indicating which penalties are active, and copies parameters from pinned host memory to device memory via the batch copy utility. Also initializes bad word tensors and end-of-sequence ID tensors for minimum length enforcement.

Forward(): Applies the active penalties in sequence on the GPU using CUDA kernels.

Usage

Used within the Generation module. Called at kSetup to configure per-batch parameters and at kForward to apply logits transformations before sampling.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/generation/logits_processor.h
• File: src/turbomind/generation/logits_processor.cc
• Lines: logits_processor.h 1-43, logits_processor.cc 1-219

Signature

class LogitsProcessor: public BaseGenerationParam {
public:
    explicit LogitsProcessor(const BaseGenerationParam& base, int phases);

    void Setup(int phase, TensorMap& env);

    void Forward(int phase, TensorMap& env);

private:
    struct Data;
    std::vector<std::shared_ptr<Data>> data_;
    std::shared_ptr<Data> buf_;  // temp host buffer
};

Import

#include "src/turbomind/generation/logits_processor.h"

I/O Contract

Inputs

Name	Type	Required	Description
base	BaseGenerationParam	Yes	Base parameters (max_batch_size, vocab_size, vocab_size_padded)
phases	int	Yes	Number of pipeline phases
env["batch"]	BatchData*	Yes (Setup)	Batch data with request caches containing generation configs
env["copy"]	BatchCopy*	Yes (Setup)	Host-to-device copy utility
env["logits"]	Tensor_<float>	Yes (Forward)	Logits tensor of shape (batch_size, vocab_size_padded)
env["token_ids_ptrs"]	Buffer_<int*>	Yes (Forward)	Pointers to token ID arrays for each request
env["sequence_length"]	Buffer_<int>	Yes (Forward)	Current sequence lengths

Outputs

Name	Type	Description
env["logits"] (modified)	Tensor_<float>	Logits with penalties applied in-place

Usage Examples

// Construction (inside Generation module)
LogitsProcessor logits_proc(base_param, phases);

// Setup: configure parameters for the current batch
logits_proc.Setup(phase, env);

// Forward: apply all active penalties to logits
logits_proc.Forward(phase, env);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime