Implementation:InternLM Lmdeploy Request

Knowledge Sources	InternLM_Lmdeploy
Domains	Inference Engine, Data Structures
Last Updated	2026-02-07 15:00 GMT

Overview

Defines the core data structures for inference requests in TurboMind, including GenerationConfig, SessionParam, Request, RequestCache, and their serialization templates.

Description

This header is the foundational data model for the TurboMind engine, defining multiple interrelated structures:

GenerationConfig: Holds all parameters controlling text generation: token limits (max_new_tokens, min_new_tokens), end-of-sequence IDs, stop/bad word IDs (as arrays of two vectors for token_id/offset pairs), sampling parameters (top_k, top_p, min_p, temperature), repetition penalty, random seed, and output control flags (output_logprobs, output_last_hidden_state, output_logits). The OutType enum defines output modes: kNone, kAll (all tokens), and kGeneration (generated tokens only).

SessionParam: Identifies a session with an ID, step counter, and lifecycle flags (start_flag, end_flag, kill_flag).

AtomicRequestState: A lock-free atomic wrapper around RequestState (status + sequence length) for thread-safe polling of request progress.

Request: The primary request object shared across all local tensor-parallel ranks. Contains the sequence and unique IDs, session params, generation config, I/O tensor maps, callbacks (end_cb, forward_cb), an atomic cancel flag, state, metrics, an error code, and optional xgrammar objects for grammar-guided decoding. Defines status codes: kOk, kInvalid, kConflict, kBusy, kInactive, kFail, kTooLong, kFinish, kCancel, kInconsistency, kNoQueue.

RequestCache: A per-rank cache for a request, holding execution state including token ID pointers, random state, sequence lengths, scheduling flags (autoregres, generating, done), cache growth parameters (alpha, beta), rope base, and output intervals.

The file also provides template functions for serialization (serdes) and for saving/loading request output tensor metadata.

Usage

Used throughout the TurboMind engine. GenerationConfig is set by callers to control generation behavior. Request objects flow through the Gateway into engine queues. RequestCache is created per-rank for each active request during batch processing.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/engine/request.h
• Lines: 1-256

Signature

struct GenerationConfig {
    int max_new_tokens = 0;
    int min_new_tokens = 0;
    std::vector<int> eos_ids;
    std::array<std::vector<int>, 2> stop_ids;
    std::array<std::vector<int>, 2> bad_ids;
    int   top_k       = 1;
    float top_p       = 0.f;
    float min_p       = 0.f;
    float temperature = 1.f;
    float repetition_penalty = 1.f;
    uint64_t random_seed = 0;
    int output_logprobs = 0;
    enum OutType { kNone = 0, kAll = 1, kGeneration = 2 };
    int output_last_hidden_state = 0;
    int output_logits            = 0;
};

struct SessionParam {
    uint64_t id;
    int step;
    bool start_flag;
    bool end_flag;
    bool kill_flag;
};

struct Request {
    uint64_t id;
    uint64_t unique_id;
    SessionParam     session;
    GenerationConfig gen_cfg;
    bool stream_output;
    TensorMap inputs;
    TensorMap outputs;
    Tensor_<int> output_ids;
    Tensor_<int> sequence_length;
    std::function<void(int)> end_cb;
    std::atomic<int> cancel_flag;
    std::function<void()> forward_cb;
    std::shared_ptr<AtomicRequestState> state;
    std::shared_ptr<RequestMetrics> metrics;
    int ec = 0;
    // Status codes: kOk(0), kInvalid(1), kConflict(2), kBusy(3), kInactive(4),
    //               kFail(5), kTooLong(6), kFinish(7), kCancel(8), kInconsistency(9), kNoQueue(10)
};

struct RequestCache {
    std::shared_ptr<Request> req;
    const Sequence*          seq;
    const GenerationConfig&  gen_cfg;
    int status = Request::kOk;
    int*     token_ids    = nullptr;
    uint8_t* random_state = nullptr;
    int step0, prompt_len, max_seq_len;
    int hidden_states_offset, logits_offset;
    int seq_len, input_len, history_len;
    bool autoregres, generating, done;
    int alpha, beta;
    float rope_base;
    Interval output_hidden_states;
    Interval output_logits;
};

Import

#include "src/turbomind/engine/request.h"

I/O Contract

Inputs

Name	Type	Required	Description
GenerationConfig fields	various	Yes	Sampling and output configuration for text generation
SessionParam fields	various	Yes	Session identification and lifecycle flags

Outputs

Name	Type	Description
Request.outputs	TensorMap	Output tensors populated during inference
RequestState (via AtomicRequestState)	RequestState	Status code and current sequence length

Usage Examples

// Configure generation parameters
GenerationConfig gen_cfg;
gen_cfg.max_new_tokens = 256;
gen_cfg.top_k = 50;
gen_cfg.top_p = 0.9f;
gen_cfg.temperature = 0.8f;

// Set up session parameters
SessionParam session;
session.id = 42;
session.start_flag = true;
session.end_flag = true;
session.kill_flag = false;
session.step = 0;

// Create a request
auto r = std::make_shared<Request>();
r->id = session.id;
r->session = session;
r->gen_cfg = gen_cfg;
r->stream_output = true;

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime