Implementation:Tencent Ncnn Benchncnn
| Knowledge Sources | |
|---|---|
| Domains | Performance, Benchmarking |
| Last Updated | 2026-02-09 19:00 GMT |
Overview
Concrete tool for measuring neural network inference performance across a suite of built-in and user-specified models using ncnn.
Description
The benchncnn utility is the standard performance measurement tool for the ncnn project. It defines a DataReaderFromEmpty class that provides zero-filled weights (since only network structure matters for speed benchmarking), loads model param data from either embedded strings or external files, runs a configurable warm-up phase (8 iterations for CPU, 10 for Vulkan GPU), then executes a timed inference loop reporting min/max/avg latency in milliseconds. When no custom model is specified, it runs a default suite of 34 models covering classification networks (SqueezeNet, MobileNet variants, ResNet, VGG16, GoogLeNet, EfficientNet, Vision Transformer), detection networks (MobileNet-SSD, MobileNet-YOLO, YOLOv4-tiny, NanoDet, YOLO-Fastest), and their int8 quantized variants. The tool supports CPU threading configuration, power-save mode (big/little cores), Vulkan GPU selection, cooling-down pauses between models, and custom model/shape parameters.
Usage
Use this tool to evaluate inference speed on any supported platform (CPU or Vulkan GPU), measure the impact of optimizations, compare configurations, and generate reproducible benchmark results. It is the tool used to produce all results documented in the ncnn benchmark README.
Code Reference
Source Location
- Repository: Tencent_Ncnn
- File: benchmark/benchncnn.cpp
- Lines: 1-467
Signature
void benchmark(const char* comment, const std::vector<ncnn::Mat>& _in, const ncnn::Option& opt, const char* model_param_data = NULL);
void benchmark(const char* comment, const ncnn::Mat& _in, const ncnn::Option& opt, const char* model_param_data = NULL);
int main(int argc, char** argv);
Import
#include "benchmark.h"
#include "cpu.h"
#include "datareader.h"
#include "net.h"
#include "gpu.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| loop_count | int (argv[1]) | No | Number of timed inference iterations (default: 4) |
| num_threads | int (argv[2]) | No | Number of CPU threads (default: big core count) |
| powersave | int (argv[3]) | No | CPU powersave mode: 0=all, 1=little, 2=big (default: 2) |
| gpu_device | int (argv[4]) | No | GPU device index, -1 for CPU-only (default: -1) |
| cooling_down | int (argv[5]) | No | Enable 10-second cooling between models (default: 1) |
| param=model.param | key=value (argv[6+]) | No | Custom model param file path |
| shape=[w,h,c],... | key=value (argv[6+]) | No | Input tensor shapes for custom model |
Outputs
| Name | Type | Description |
|---|---|---|
| stderr output | text | Per-model timing: min, max, avg latency in milliseconds |
| exit code | int | 0 on success, -1 on error |
Usage Examples
Running the Default Benchmark Suite
# Run with 4 loops, 4 threads, big cores, CPU-only, cooling enabled
./benchncnn 4 4 2 -1 1
Running with Vulkan GPU
# Run on GPU device 0
./benchncnn 8 1 0 0 1
Benchmarking a Custom Model
./benchncnn 4 4 2 -1 1 param=mymodel.param shape=[224,224,3]
Key Code Pattern
ncnn::Net net;
net.opt = opt;
net.load_param(comment);
DataReaderFromEmpty dr;
net.load_model(dr);
const std::vector<const char*>& input_names = net.input_names();
const std::vector<const char*>& output_names = net.output_names();
for (int i = 0; i < g_loop_count; i++)
{
double start = ncnn::get_current_time();
{
ncnn::Extractor ex = net.create_extractor();
for (size_t j = 0; j < input_names.size(); ++j)
{
ncnn::Mat in = _in[j];
ex.input(input_names[j], in);
}
for (size_t j = 0; j < output_names.size(); ++j)
{
ncnn::Mat out;
ex.extract(output_names[j], out);
}
}
double end = ncnn::get_current_time();
double time = end - start;
time_min = std::min(time_min, time);
time_max = std::max(time_max, time);
time_avg += time;
}
time_avg /= g_loop_count;
fprintf(stderr, "%20s min = %7.2f max = %7.2f avg = %7.2f\n", comment, time_min, time_max, time_avg);