Principle:Ggml org Llama cpp Model Inspection
| Field | Value |
|---|---|
| Principle Name | Model Inspection |
| Category | Pre-Conversion Analysis |
| Scope | Examining model architecture and tensor structure before conversion |
| Status | Active |
Overview
Description
Before converting a model from one format to another, it is essential to inspect the model's internal structure. Model inspection answers critical questions that inform the conversion process:
- What tensors are present? The tensor inventory reveals the model's architecture (number of layers, attention heads, hidden dimensions) and identifies any unexpected or non-standard tensors.
- What are the tensor shapes? Shape information determines memory requirements, validates that the model matches its configuration file, and identifies potential issues with tensor concatenation or splitting.
- What data types are used? The storage dtype (float16, bfloat16, float32) affects both the conversion output type and the numerical fidelity of the result.
- Is the model stored as a single file or multiple shards? Sharded models require special handling during loading, and the shard structure must be understood before conversion begins.
Inspection serves as a diagnostic checkpoint in the conversion pipeline. By examining the model before conversion, operators can detect problems early: missing tensors, corrupted files, architecture mismatches, or unexpected dtypes that would cause silent failures or incorrect outputs later.
Usage
Model inspection is typically performed immediately after model acquisition and before conversion. The inspection workflow is:
- Point the inspection tool at the model directory
- Review the tensor inventory: names, shapes, and dtypes
- Compare against expected architecture parameters from
config.json - Identify any anomalies (missing layers, unexpected dtypes, extra tensors)
- Proceed to conversion if the inspection results are satisfactory
Theoretical Basis
Model inspection draws on the principle of pre-condition verification from software engineering. Just as a function should validate its inputs before proceeding, a conversion pipeline should validate the model structure before attempting transformation.
Tensor naming conventions in HuggingFace models follow a hierarchical pattern that encodes architectural information:
model.layers.{layer_idx}.self_attn.q_proj.weight # Query projection, layer N
model.layers.{layer_idx}.self_attn.k_proj.weight # Key projection, layer N
model.layers.{layer_idx}.self_attn.v_proj.weight # Value projection, layer N
model.layers.{layer_idx}.mlp.gate_proj.weight # MLP gate projection
model.embed_tokens.weight # Token embedding matrix
lm_head.weight # Output projection
By parsing these names, inspection tools can reconstruct the full architecture: number of transformer blocks, attention mechanism type (MHA, GQA, MQA), MLP variant (standard, gated), and whether the model uses tied embeddings.
Shape analysis provides a second layer of verification. For a model with hidden dimension d, n_heads attention heads, and head dimension d_head:
- Query projections should have shape
[n_heads * d_head, d] - Key/value projections should have shape
[n_kv_heads * d_head, d]for grouped-query attention - MLP projections should have shape
[intermediate_size, d]or its transpose
Mismatches between these expected shapes and the actual tensor shapes indicate configuration errors or model corruption.
Data type analysis informs the --outtype parameter selection. If the source model uses bfloat16 storage, converting with --outtype bf16 preserves full fidelity, while --outtype f16 involves a dtype cast that may introduce small numerical differences.