Implementation:Ggml org Llama cpp CLIP Impl
| Knowledge Sources | |
|---|---|
| Domains | Multimodal, Vision |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Internal implementation header for the CLIP module, defining core data structures, GGUF metadata constants, and enumerations shared across all CLIP implementation files.
Description
Defines GGUF key constants for model metadata (embedding dimensions, layer counts, image sizes, patch sizes, normalization parameters) and tensor name templates for weight loading. Declares enumerations for projector types (MLP, LDP, LDPv2, Pixtral, etc.) and internal image data structures (`clip_image_u8`, `clip_image_f32`, `clip_image_u8_batch`, `clip_image_f32_batch`). Includes the `clip_ctx` struct (the main CLIP context) and various RAII deleters for image types.
Usage
Use this header when implementing or extending CLIP model backends within the multimodal subsystem. It provides the shared type definitions and constants required by clip.cpp, clip-graph.h, and model-specific backends.
Code Reference
Source Location
- Repository: Ggml_org_Llama_cpp
- File: tools/mtmd/clip-impl.h
- Lines: 1-584
Signature
// GGUF key constants
#define KEY_N_EMBD "clip.%s.embedding_length"
#define KEY_N_FF "clip.%s.feed_forward_length"
#define KEY_N_BLOCK "clip.%s.block_count"
#define KEY_N_HEAD "clip.%s.attention.head_count"
#define KEY_IMAGE_SIZE "clip.vision.image_size"
#define KEY_PATCH_SIZE "clip.vision.patch_size"
// Tensor name templates
#define TN_POS_EMBD "%s.position_embd.weight"
#define TN_PATCH_EMBD "v.patch_embd.weight"
#define TN_ATTN_QKV "%s.blk.%d.attn_qkv.%s"
// Core data structures
struct clip_ctx;
struct clip_image_u8;
struct clip_image_f32;
struct clip_image_u8_batch;
struct clip_image_f32_batch;
Import
#include "ggml.h"
#include "gguf.h"
#include "clip.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| GGUF model file | file | Yes | GGUF file containing CLIP vision model weights and metadata |
| image data | clip_image_u8 / clip_image_f32 | Yes | Raw or preprocessed image pixel data |
Outputs
| Name | Type | Description |
|---|---|---|
| clip_ctx | struct | Initialized CLIP context with loaded model weights, hyperparameters, and compute graph |
| clip_image_f32 | struct | Preprocessed floating-point image data ready for inference |
Usage Examples
// Internal usage - typically included by clip.cpp
#include "clip-impl.h"
// Access GGUF keys for model metadata
char key_buf[256];
snprintf(key_buf, sizeof(key_buf), KEY_N_EMBD, "vision");
// Work with image data structures
clip_image_u8 img;
img.nx = width;
img.ny = height;
img.buf.resize(width * height * 3);