Implementation:InternLM Lmdeploy ModelRequest

Knowledge Sources	InternLM_Lmdeploy
Domains	Inference Engine, API
Last Updated	2026-02-07 15:00 GMT

Overview

Provides the ModelRequest class, which serves as the public API for submitting inference requests to the TurboMind engine, handling forward passes, cancellation, session termination, and grammar-guided decoding.

Description

The ModelRequest class encapsulates the full lifecycle of an inference request from the caller's perspective. It holds a pointer to the Gateway for request submission, and stores model metadata (data type, session length, vocab size, hidden dimension) needed to correctly allocate output buffers.

InputParam struct: Groups the inputs for a forward call, including a shared TensorMap of input tensors, session parameters, generation configuration, and flags for stream output and metrics.

OutputParam struct: Returns from Forward() with a shared TensorMap of output tensors, an atomic request state for polling completion, and optional request metrics.

Key methods:

• Forward(): The main entry point for inference. It validates input dimensions, allocates output buffers (output_ids, sequence_length, and optional logits/hidden_state/logprob buffers based on GenerationConfig), constructs a Request object, optionally attaches a grammar matcher, stores a weak reference for cancellation, and pushes the request to the Gateway.
• Cancel(): Attempts to cancel an in-flight request by locking the weak reference and forwarding to the Gateway's cancel method.
• End(): Terminates a session by sending a kill request with the given session ID.
• setGrammar(): Attaches a compiled xgrammar grammar for grammar-guided decoding in subsequent forward calls.

Usage

Created by TurboMind::CreateRequest(). Callers use Forward() to submit inputs and receive an OutputParam with results and state. Use Cancel() to abort a running request, End() to terminate a session, and setGrammar() to enable structured output generation.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/engine/model_request.h
• File: src/turbomind/engine/model_request.cc
• Lines: model_request.h 1-67, model_request.cc 1-151

Signature

class ModelRequest {
public:
    virtual ~ModelRequest() = default;

    ModelRequest(Gateway* gateway, DataType data_type, int session_len, int vocab_size, int hidden_dim);

    void Cancel();

    void End(std::function<void(int)> cb, uint64_t session_id);

    struct InputParam {
        std::shared_ptr<TensorMap> tensors;
        SessionParam     session;
        GenerationConfig gen_cfg;
        bool stream_output;
        bool enable_metrics;
    };

    struct OutputParam {
        std::shared_ptr<TensorMap>          tensors;
        std::shared_ptr<AtomicRequestState> state;
        std::shared_ptr<RequestMetrics>     metrics;
    };

    OutputParam Forward(InputParam param, std::function<void()> cb);

    void setGrammar(const xgrammar::CompiledGrammar& grammar);

protected:
    Gateway* const gateway_;
    const DataType data_type_;
    const int session_len_;
    const int hidden_dim_;
    const int vocab_size_;
    uint64_t session_id_;
    std::weak_ptr<Request> request_;
    std::shared_ptr<TensorMap> inputs_;
    std::shared_ptr<TensorMap> outputs_;
    std::shared_ptr<xgrammar::CompiledGrammar> grammar_;
};

Import

#include "src/turbomind/engine/model_request.h"

I/O Contract

Inputs

Name	Type	Required	Description
param.tensors	std::shared_ptr<TensorMap>	Yes	Input tensors including "input_ids" (1D)
param.session	SessionParam	Yes	Session ID, step, start/end/kill flags
param.gen_cfg	GenerationConfig	Yes	Generation parameters (max_new_tokens, top_k, temperature, etc.)
param.stream_output	bool	Yes	Whether to stream partial outputs
param.enable_metrics	bool	Yes	Whether to track request metrics
cb	std::function<void()>	Yes	Callback invoked on forward completion

Outputs

Name	Type	Description
tensors	std::shared_ptr<TensorMap>	Output tensors: "output_ids", "sequence_length", optionally "logits", "last_hidden_state", "logprob_vals", "logprob_indexes", "logprob_nums"
state	std::shared_ptr<AtomicRequestState>	Atomic state for polling request status and sequence length
metrics	std::shared_ptr<RequestMetrics>	Request timing metrics (null if metrics disabled)

Usage Examples

// Create a model request
auto request = turbomind_instance.CreateRequest();

// Prepare input
ModelRequest::InputParam param;
param.tensors = std::make_shared<TensorMap>();
param.tensors->emplace("input_ids", input_ids_tensor);
param.session.id = 1;
param.session.start_flag = true;
param.session.end_flag = true;
param.gen_cfg.max_new_tokens = 128;
param.gen_cfg.top_k = 1;
param.stream_output = false;
param.enable_metrics = true;

// Submit for inference
auto output = request->Forward(param, []() { /* completion callback */ });

// Poll for completion
// output.state can be checked for status updates

// Cancel if needed
request->Cancel();

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime