Implementation:Mlc ai Mlc llm Image Utils

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	C++, Image Processing, Vision LLM, CLIP
Last Updated	2026-02-09 19:00 GMT

Overview

image_utils.cc implements image loading from base64-encoded strings and CLIP-compatible image preprocessing (resize, center crop, rescale, normalize, and channel reordering) for multimodal LLM inference in MLC LLM.

Description

This implementation file within the mlc::llm::json_ffi namespace provides two key functions for handling image data in vision-language models:

MemoryBufferStream is a helper class extending dmlc::Stream that wraps an in-memory buffer as a read-only stream. It is used internally to feed base64-encoded data into TVM's base64 decoder. The Write method is intentionally unsupported and triggers a fatal error.

Base64DecodedSize calculates the expected decoded size of a base64-encoded string by examining trailing padding characters (=).

LoadImageFromBase64 decodes a base64-encoded image string into a TVM tensor:

1. Creates a MemoryBufferStream from the base64 string
2. Wraps it in a tvm::support::Base64InStream for decoding
3. Decodes the binary data into a byte buffer
4. Uses stb_image (stbi_load_from_memory) to parse the image as RGB (3 channels)
5. Creates a CPU tensor of shape [height, width, 3] with uint8 dtype
6. Returns Result::Ok on success or Result::Error with the stb failure reason

ClipPreprocessor transforms a raw image tensor into the format expected by CLIP vision encoders:

1. Resize -- Resizes the image so the short side matches target_size while maintaining aspect ratio, using bilinear interpolation
2. Center crop -- Crops the center target_size x target_size region from the resized image
3. Rescale -- Divides all pixel values by 255.0 to normalize to [0, 1]
4. Normalize -- Applies CLIP standard normalization with mean [0.48145466, 0.4578275, 0.40821073] and std [0.26862954, 0.26130258, 0.27577711]
5. Channel reorder -- Transposes from HWC (height, width, channels) to CHW (channels, height, width) format
6. Tensor creation -- Creates a float32 tensor of shape [1, 3, target_size, target_size] on the specified device

Usage

These utilities are called during multimodal chat processing when the user provides an image (via base64 encoding in the message content). LoadImageFromBase64 decodes the image data, and ClipPreprocessor prepares it for the vision encoder component of a vision-language model.

Code Reference

Source Location

• Repository: Mlc_ai_Mlc_llm
• File: cpp/json_ffi/image_utils.cc

Signature

namespace mlc {
namespace llm {
namespace json_ffi {

// Helper: read-only memory stream for base64 decoding
class MemoryBufferStream : public dmlc::Stream {
 public:
  MemoryBufferStream(const char* data, size_t size);
  size_t Read(void* ptr, size_t size) override;
  size_t Write(const void* ptr, size_t size) override;  // unsupported
};

// Calculate decoded size of a base64 string
size_t Base64DecodedSize(const std::string& base64_str);

// Decode a base64-encoded image into a TVM tensor
Result<Tensor> LoadImageFromBase64(const std::string& base64_str);

// Preprocess an image tensor for CLIP vision encoder
Tensor ClipPreprocessor(Tensor image_data, int target_size, DLDevice device);

}  // namespace json_ffi
}  // namespace llm
}  // namespace mlc

Import

#include "image_utils.h"

I/O Contract

Function	Input	Type	Description
LoadImageFromBase64	base64_str	const std::string&	Base64-encoded image data (JPEG, PNG, etc.)
ClipPreprocessor	image_data	Tensor	Raw image tensor of shape [H, W, 3] with uint8 values
ClipPreprocessor	target_size	int	Target spatial dimension (e.g., 224 for CLIP ViT-B)
ClipPreprocessor	device	DLDevice	Target device for the output tensor (e.g., GPU)

Function	Output	Type	Description
LoadImageFromBase64	Decoded image	Result<Tensor>	CPU tensor of shape [H, W, 3], uint8, or an error with stb failure reason
ClipPreprocessor	Preprocessed image	Tensor	Float32 tensor of shape [1, 3, target_size, target_size] on the specified device, normalized with CLIP statistics

Preprocessing Step	Details
Resize	Bilinear interpolation; short side scaled to target_size, aspect ratio preserved
Center crop	Extracts target_size x target_size from the center of the resized image
Rescale	Pixel values divided by 255.0
Normalize	Mean: [0.48145466, 0.4578275, 0.40821073], Std: [0.26862954, 0.26130258, 0.27577711]
Channel reorder	HWC to CHW layout, with batch dimension prepended

Usage Examples

#include "image_utils.h"

// Load an image from base64 string
std::string base64_image = "...";  // base64-encoded JPEG/PNG
auto result = LoadImageFromBase64(base64_image);
if (result.IsOk()) {
    Tensor raw_image = result.Unwrap();

    // Preprocess for CLIP with 224x224 target size
    DLDevice device = {kDLOpenCL, 0};
    Tensor processed = ClipPreprocessor(raw_image, 224, device);
    // processed shape: [1, 3, 224, 224], float32, normalized
} else {
    LOG(ERROR) << "Failed to load image: " << result.GetError();
}