Implementation:InternLM Lmdeploy Generation

Knowledge Sources	InternLM_Lmdeploy
Domains	Text Generation, Inference Engine
Last Updated	2026-02-07 15:00 GMT

Overview

Provides the top-level Generation class that orchestrates all token generation operations (logits processing, sampling, stop criteria, guided decoding) during inference batch execution.

Description

The Generation class serves as the unified interface for the generation pipeline in TurboMind. It aggregates all generation-related sub-modules (LogitsProcessor, Sampling, StopCriteria, GuidedDecoding) behind a single Run() method that dispatches to the appropriate sub-module based on the BatchOp operation type and phase.

The class uses the pimpl idiom with a private Impl struct. It is constructed with the model's data type, maximum batch size, session length, vocabulary sizes (raw and padded), a tensor-parallel host communicator, and the number of pipeline phases.

The Run() method accepts a BatchOp, a phase index, and a TensorMap environment, enabling it to be called at different points in the batch processing lifecycle (setup, forward, fetch, update).

Usage

Instantiated by the engine during initialization. Called during each batch processing cycle at the appropriate lifecycle stages (kSetup, kForward, kFetch, kUpdate) to apply logits transformations, perform token sampling, check stop criteria, and enforce grammar constraints.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/generation/generation.h
• Lines: 1-38

Signature

class Generation {
public:
    ~Generation();

    Generation(DataType              data_type,
               int                   max_batch_size,
               int                   session_len,
               int                   vocab_size,
               int                   vocab_size_padded,
               const comm::HostComm& tp_group,
               int                   phases);

    void Run(BatchOp op, int phase, TensorMap& env);

private:
    struct Impl;
    std::unique_ptr<Impl> impl_;
};

Import

#include "src/turbomind/generation/generation.h"

I/O Contract

Inputs

Name	Type	Required	Description
data_type	DataType	Yes	Compute data type (e.g., float16)
max_batch_size	int	Yes	Maximum number of requests in a batch
session_len	int	Yes	Maximum session/sequence length
vocab_size	int	Yes	Raw vocabulary size
vocab_size_padded	int	Yes	Padded vocabulary size (for GPU alignment)
tp_group	const comm::HostComm&	Yes	Tensor-parallel communication group
phases	int	Yes	Number of pipeline phases
op (Run)	BatchOp	Yes	The batch operation to execute
phase (Run)	int	Yes	Pipeline phase index
env (Run)	TensorMap&	Yes	Environment tensor map with batch data, logits, etc.

Outputs

Name	Type	Description
env (modified)	TensorMap&	Environment is modified in place with sampled tokens, stop flags, etc.

Usage Examples

// Create the generation module
Generation gen(data_type, max_batch_size, session_len, vocab_size, vocab_size_padded, tp_group, phases);

// During batch processing:
// Setup phase - initialize sampling parameters, stop words, etc.
gen.Run(BatchOp::kSetup, phase, env);

// Forward phase - apply logits processing, sampling, stop criteria
gen.Run(BatchOp::kForward, phase, env);

// Fetch phase - copy sampling results back to host
gen.Run(BatchOp::kFetch, phase, env);

// Update phase - update grammar matchers, copy logprobs
gen.Run(BatchOp::kUpdate, phase, env);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime