Implementation:InternLM Lmdeploy Gemm Cast

Knowledge Sources	InternLM_Lmdeploy
Domains	GPU_Kernels, GEMM
Last Updated	2026-02-07 15:00 GMT

Overview

Provides host-callable CUDA kernel wrappers for data type conversion operations including 4-bit/8-bit extension, compaction, transposition, scale/zero fusion, and output dimension interleaving.

Description

This header declares a set of utility functions for weight preprocessing and quantization data manipulation:

• extend_to_u8: Extends 4-bit (uint4_t) packed values to 8-bit (uint8_t)
• extend_to_u16: Extends 4-bit or 8-bit values to 16-bit
• compact_to_u4: Compacts 8-bit values to 4-bit packed format
• transpose_u4: Transposes a matrix of 4-bit values
• fuse_scales_and_zeros: Merges separate scale and zero-point arrays into a fused FP16 array
• interleave_output_dims: Interleaves two output dimension arrays for gated activation support
• AdjustUe8m0ScaleForHalf: Adjusts UE8M0 block-scale factors for FP16 MMA compatibility
• BlockscaleToGroupscale: Converts block-scale format to group-scale format

All functions accept cudaStream_t for async execution.

Usage

Used during weight loading and preprocessing to convert quantized model weights into the format expected by GEMM kernels.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: src/turbomind/kernels/gemm/cast.h

Signature

void extend_to_u8(uint8_t* dst, const uint4_t* src, size_t n, cudaStream_t st = {});
void extend_to_u16(uint16_t* dst, const uint4_t* src, size_t n, cudaStream_t st = {});
void compact_to_u4(uint4_t* dst, const uint8_t* src, size_t n, cudaStream_t st = {});
void transpose_u4(uint4_t* dst, const uint4_t* src, int s, int c, cudaStream_t st = {});
void fuse_scales_and_zeros(half* fused, const half* scales, half* zeros, size_t n, cudaStream_t st = {});
template<class T> void interleave_output_dims(T* fused, const T* a, const T* b, int m, int k, cudaStream_t st);
void AdjustUe8m0ScaleForHalf(uint8_t* data, int n, cudaStream_t st);
Tensor BlockscaleToGroupscale(const Tensor& scales, DataType data_type, int block_size);

Import

#include "src/turbomind/kernels/gemm/cast.h"

I/O Contract

Inputs

Name	Type	Required	Description
src	const void*	Yes	Source data in original format
n	size_t	Yes	Number of elements
st	cudaStream_t	No	CUDA stream (default: 0)

Outputs

Name	Type	Description
dst	void*	Converted data in target format

Usage Examples

extend_to_u8(u8_weights, u4_weights, num_elements, stream);
fuse_scales_and_zeros(fused_sq, scales, zeros, group_count, stream);

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime