Implementation:InternLM Lmdeploy InputProcessor

Knowledge Sources	InternLM_Lmdeploy
Domains	Inference Engine, Input Processing
Last Updated	2026-02-07 15:00 GMT

Overview

Processes and prepares input data for the language model forward pass, handling token ID assembly, input embedding patches, and autoregressive token injection across batch operations.

Description

The InputProcessor class manages the transformation of raw request inputs into the format required by the language model. It uses the pimpl idiom with a private Impl struct that contains the full implementation.

The processor operates across three batch lifecycle phases:

Add (kAdd): Validates and processes newly added requests. Trims input embeddings based on the sequence's current token range, and handles input_embedding_ranges and input_embeddings tensors from the request. Validates embedding ranges for sorting, bounds, and size correctness, returning error codes (e.g., Request::kInvalid) on failure. Clones embedding data for persistent sessions.

Setup (kSetup): Prepares the input token arrays for the current batch. For each request, it either copies the full token sequence (for prefill/context requests with autoregres == false) or reserves a single slot for the autoregressive token. Builds offset arrays for variable-length batched inputs, copies data from pinned host memory to device, computes total token counts, and prepares input embedding coordinates for patching.

Prepare (kPrepare): Injects autoregressive tokens into the input array. For requests in autoregressive mode, copies the last output token from the model's autoregressive ID buffer into the reserved input slot. Produces the final input_ids, q_offsets, and selected_token_pos tensors into the environment.

PatchEmbedding(): A separate method called during the model forward pass to overwrite portions of the computed word embeddings with custom input embeddings (for multimodal or prefix-tuning scenarios).

The Data struct per phase holds device-side buffers for input IDs, offsets, selected token positions, autoregressive ID positions, and input embedding coordinates.

Usage

Instantiated during engine initialization with engine and model parameters. Called by the engine at kAdd, kSetup, and kPrepare stages. PatchEmbedding is called separately by the language model during its forward pass.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/models/input_processor.h
• File: src/turbomind/models/input_processor.cc
• Lines: input_processor.h 1-23, input_processor.cc 1-268

Signature

class InputProcessor {
public:
    ~InputProcessor();

    InputProcessor(const EngineParam& engine, const ModelParam& model, int phases);

    void Run(BatchOp op, int phase, TensorMap& env);

    void PatchEmbedding(int phase, Tensor& embeds, BatchCopy& copy);

private:
    struct Impl;
    std::unique_ptr<Impl> impl_;
};

Import

#include "src/turbomind/models/input_processor.h"

I/O Contract

Inputs

Name	Type	Required	Description
engine	EngineParam	Yes	Engine parameters (max_batch_size, max_forward_token_num)
model	ModelParam	Yes	Model parameters (hidden_units, data_type)
phases	int	Yes	Number of pipeline phases
env["requests"]	Buffer_<RequestCache*>	Yes (Add)	New requests to validate and process
env["batch"]	BatchData*	Yes (Setup/Prepare)	Current batch data with request caches
env["copy"]	BatchCopy*	Yes (Setup/Prepare)	Host-to-device copy utility
env["autoreg_ids"]	Buffer_<int>	Yes (Prepare)	Autoregressive token IDs from previous step

Outputs

Name	Type	Description
env["token_num"]	Buffer	Total number of input tokens in the batch
env["input_ids"]	Buffer_<int>	Flattened input token IDs for the batch
env["q_offsets"]	Buffer_<int>	Per-request offsets into the input_ids array (length: bsz+1)
env["selected_token_pos"]	Buffer_<int>	Position of the last token for each request (for output extraction)

Usage Examples

// Construction
InputProcessor input_proc(engine_param, model_param, phases);

// During batch processing:
input_proc.Run(BatchOp::kAdd, phase, env);     // Validate new requests
input_proc.Run(BatchOp::kSetup, phase, env);   // Prepare input arrays
input_proc.Run(BatchOp::kPrepare, phase, env); // Inject autoregressive tokens

// During model forward pass:
input_proc.PatchEmbedding(phase, embedding_tensor, copy);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime