Implementation:InternLM Lmdeploy Gemm Convert

Knowledge Sources	InternLM_Lmdeploy
Domains	GPU_Kernels, GEMM
Last Updated	2026-02-07 15:00 GMT

Overview

Defines the LayoutConverter interface for converting weight matrix layouts between formats, along with helpers for creating strided and blocked pointer arrays on the GPU.

Description

This header provides the abstraction for weight layout conversion needed when model weights must be repacked for a specific GEMM kernel's expected memory layout:

• LayoutConverter: Abstract base struct with a virtual Convert method that transforms a source matrix (S with Sdesc) into a destination matrix (D with Ddesc) on the given CUDA stream. Each converter is associated with a target Order and Pack format.

• GetConverters: Factory function that returns an array<const LayoutConverter*, 2> (for the two operands) based on data types (data, weight, input), grouped flag, and SM architecture.

• MakeStridedPtrs: Allocates GPU memory and copies a vector of (pointer, stride) pairs for strided batch access.

• MakeBlockedPtrs: Allocates GPU memory and copies a vector of (pointer, offset) pairs for blocked batch access.

Usage

Called during model weight initialization to convert weights from the stored format to the kernel-optimized format.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/kernels/gemm/convert.h

Signature

struct LayoutConverter {
    Order order;
    Pack  pack;
    virtual int Convert(const void* S, const MatrixLayout& Sdesc,
                        void* D, MatrixLayout& Ddesc, cudaStream_t stream) const = 0;
};

std::array<const LayoutConverter*, 2> GetConverters(DataType data_type, DataType weight_type,
                                                     DataType input_type, bool grouped, int sm);

void* MakeStridedPtrs(const std::vector<std::pair<void*, int>>& ptrs, cudaStream_t stream);
void* MakeBlockedPtrs(const std::vector<std::pair<void*, int>>& ptrs, cudaStream_t stream);

Import

#include "src/turbomind/kernels/gemm/convert.h"

I/O Contract

Inputs

Name	Type	Required	Description
S	const void*	Yes	Source weight data
Sdesc	MatrixLayout	Yes	Source layout descriptor
data_type, weight_type, input_type	DataType	Yes	Data type configuration
sm	int	Yes	GPU architecture (e.g., 80, 90)

Outputs

Name	Type	Description
D	void*	Converted weight data in target layout
Ddesc	MatrixLayout	Updated layout descriptor for the target format

Usage Examples

auto converters = GetConverters(DataType::kHalf, DataType::kInt4, DataType::kHalf, false, 80);
converters[0]->Convert(src_weights, src_desc, dst_weights, dst_desc, stream);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime