Implementation:Ggml org Llama cpp Check NMSE

Knowledge Sources	Ggml_org_Llama_cpp
Domains	Model_Conversion, Verification
Last Updated	2026-02-15 00:00 GMT

Overview

Validates model conversion quality by computing Normalized Mean Squared Error (NMSE) between PyTorch and llama.cpp logits.

Description

Loads reference (PyTorch) and test (llama.cpp) logits from binary files, verifies shape compatibility, then computes NMSE, MSE, max/mean absolute error, and NMSE in dB. Provides an interpretation scale from "Essentially identical" (NMSE < 1e-6) to "Very poor match" (NMSE > 1.0). Exits with code 0 if NMSE < 1e-2 (pass) or 1 otherwise.

Usage

Use this script for quantitative pass/fail checks of model conversion accuracy, enabling automated CI/CD validation of GGUF conversions.

Code Reference

Source Location

• Repository: Ggml_org_Llama_cpp
• File: examples/model-conversion/scripts/utils/check-nmse.py
• Lines: 1-177

Signature

def calculate_nmse(reference, test)
def load_logits(file_path)
def interpret_nmse(nmse)
def main()

Import

import numpy as np
import sys
import os
import argparse
from pathlib import Path
from common import get_model_name_from_env_path

I/O Contract

Inputs

Name	Type	Required	Description
-m / --model-path	str	Yes	Path to the model directory
MODEL_PATH	env var	Yes	Environment variable pointing to the PyTorch model path
CONVERTED_MODEL	env var	Yes	Environment variable pointing to the converted llama.cpp model path
pytorch-{name}.bin	file	Yes	Binary file containing reference PyTorch logits
llamacpp-{name}.bin	file	Yes	Binary file containing test llama.cpp logits

Outputs

Name	Type	Description
stdout	text	NMSE value, interpretation, MSE, max/mean absolute error, and NMSE in dB
exit code	int	0 if NMSE < 1e-2 (pass), 1 otherwise (fail)

Usage Examples

# Run NMSE check from the model conversion scripts directory
export MODEL_PATH=/path/to/pytorch/model
export CONVERTED_MODEL=/path/to/converted/model.gguf

python check-nmse.py -m /path/to/model

Related Pages

• Principle:Ggml_org_Llama_cpp_Model_Conversion