Implementation:InternLM Lmdeploy Gemm Interface

Knowledge Sources	InternLM_Lmdeploy
Domains	GPU_Kernels, GEMM
Last Updated	2026-02-07 15:00 GMT

Overview

The top-level Gemm class providing a unified GEMM interface with automatic kernel selection, caching, tuning, and serialization of dispatch results.

Description

The Gemm class is the primary entry point for invoking GEMM operations in the TurboMind engine. It uses a PIMPL pattern to encapsulate:

• Kernel registry: All registered GEMM kernels for the current device
• Dispatch cache: Cached tuning results for previously-seen problem sizes
• Context: Problem analysis and kernel filtering
• Measurer: Performance measurement for autotuning

Key methods:

• Run: Executes D = alpha * (A @ B) + beta * C with full operand support (A, U, B, V, C, D), handling kernel selection, caching, and optional autotuning
• Export/Import: Serializes/deserializes dispatch cache for persistence
• GetTuningSeq: Returns the batch-size tuning sequence

Workspace requirements: kBarriersSize (1 MB) for split-K barriers and kPartialsSize (32 MB) for partial result accumulation.

Usage

Created once per model/device, used for all GEMM operations (linear layers, MoE projections) throughout inference.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/kernels/gemm/gemm.h

Signature

class Gemm {
public:
    static constexpr size_t kBarriersSize = 1 << 20;
    static constexpr size_t kPartialsSize = 32 << 20;

    Gemm();
    ~Gemm();

    [[nodiscard]] int Run(const Operation& operation, float alpha,
                          const void* A, const MatrixLayout& Adesc,
                          const void* U, const MatrixLayout& Udesc,
                          const void* B, const MatrixLayout& Bdesc,
                          const void* V, const MatrixLayout& Vdesc,
                          float beta,
                          const void* C, const MatrixLayout& Cdesc,
                          void* D, const MatrixLayout& Ddesc,
                          const Workspace& workspace, cudaStream_t stream);

    int Export(std::ostream& os);
    int Import(std::istream& is);
    std::vector<int> GetTuningSeq() const;
};

Import

#include "src/turbomind/kernels/gemm/gemm.h"

I/O Contract

Inputs

Name	Type	Required	Description
operation	Operation	Yes	Dispatch policy, epilogue, quantization config
alpha, beta	float	Yes	GEMM scaling factors
A, B	const void*	Yes	Input matrix operands
U, V	const void*	No	Quantization scale operands
C	const void*	No	Accumulation input (for beta != 0)
workspace	Workspace	Yes	Barriers and partials buffers

Outputs

Name	Type	Description
D	void*	GEMM output matrix
return	int	0 on success

Usage Examples

Gemm gemm;
gemm.Import(cache_file);
int status = gemm.Run(operation, 1.0f,
    A, Adesc, U, Udesc, B, Bdesc, V, Vdesc,
    0.0f, nullptr, Cdesc, D, Ddesc, workspace, stream);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime