Implementation:InternLM Lmdeploy ModelExecutor

Knowledge Sources	InternLM_Lmdeploy
Domains	Inference Engine, Model Execution
Last Updated	2026-02-07 15:00 GMT

Overview

Provides the ModelExecutor class that runs the auto-regressive language model forward pass on batched data, consuming from an inbound queue and producing to an outbound queue.

Description

The ModelExecutor class implements the device-side execution loop for auto-regressive language models in TurboMind. It uses the pimpl idiom to hide its implementation details behind a private Impl struct.

The executor sits between two Queue<std::unique_ptr<BatchData>> channels: an inbound queue from which it receives prepared batch data, and an outbound queue to which it sends completed batch data. This producer-consumer pattern decouples scheduling from execution, allowing the engine to prepare the next batch while the current one is being executed on the GPU.

The class is move-only (non-copyable) and provides an explicit operator bool() for initialization checks. After construction, Start() launches the execution thread that continuously processes batches.

Usage

Created internally by the Engine during initialization. The Engine feeds prepared BatchData into the inbound queue, and the ModelExecutor processes them by running the LanguageModel's forward pass, then places results on the outbound queue for post-processing.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/engine/model_executor.h
• Lines: 1-42

Signature

class ModelExecutor {
public:
    ~ModelExecutor();
    ModelExecutor();
    ModelExecutor(ModelExecutor&&) noexcept;
    ModelExecutor& operator=(ModelExecutor&&) noexcept;

    explicit operator bool() const noexcept;

    ModelExecutor(LanguageModel&                     model,
                  Context&                           context,
                  int                                device_id,
                  Queue<std::unique_ptr<BatchData>>& inbound,
                  Queue<std::unique_ptr<BatchData>>& outbound);

    void Start();

private:
    struct Impl;
    std::unique_ptr<Impl> impl_;
};

Import

#include "src/turbomind/engine/model_executor.h"

I/O Contract

Inputs

Name	Type	Required	Description
model	LanguageModel&	Yes	Reference to the language model for forward pass execution
context	Context&	Yes	Shared execution context (CUDA streams, memory pools)
device_id	int	Yes	CUDA device index for this executor
inbound	Queue<std::unique_ptr<BatchData>>&	Yes	Queue from which prepared batches are consumed
outbound	Queue<std::unique_ptr<BatchData>>&	Yes	Queue to which completed batches are produced

Outputs

Name	Type	Description
BatchData (via outbound queue)	std::unique_ptr<BatchData>	Completed batch data with model forward pass results

Usage Examples

// Create the executor connected to inbound/outbound queues
ModelExecutor executor(model, context, device_id, inbound_queue, outbound_queue);

// Start the execution loop in a background thread
executor.Start();

// The engine pushes prepared batches to the inbound queue
// and retrieves results from the outbound queue

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime