Workflow:Alibaba MNN Python Model Inference
| Knowledge Sources | |
|---|---|
| Domains | Deep_Learning, Model_Inference, On_Device_AI |
| Last Updated | 2026-02-10 08:00 GMT |
Overview
End-to-end process for loading an MNN model in Python and performing inference with configurable hardware backends (CPU, GPU via OpenCL/Metal/Vulkan/CUDA, NPU).
Description
This workflow covers the standard procedure for running neural network inference using MNN's Python API (PyMNN). It demonstrates data preprocessing with MNN.cv and MNN.numpy (lightweight OpenCV and NumPy replacements), model loading via the nn.Module API, forward pass execution, and output post-processing. The workflow supports multiple hardware backends and precision modes (FP32, FP16, INT8 dynamic quantization) through RuntimeManager configuration.
Key outputs:
- Model inference results as MNN expr.VARP tensors
- Configurable backend selection (CPU, GPU, NPU) for optimal performance per device
Usage
Execute this workflow when you have an MNN-format model (.mnn file, obtained via the Model Conversion Pipeline) and need to run inference from Python. This is the primary workflow for ML engineers integrating MNN models into Python applications, prototyping inference pipelines, or benchmarking model performance across different hardware backends.
Execution Steps
Step 1: Install PyMNN
Install the MNN Python package which provides the inference runtime, conversion tools, and lightweight CV/NumPy libraries. The package can be installed via pip or compiled from source for custom configurations.
Key considerations:
- pip install MNN is the quickest route
- If pip install fails for the current platform, compile from source: cd pymnn/pip_package && python3 build_deps.py && python3 setup.py install
- The build_deps.py script accepts optional arguments like "llm" to enable LLM support
Step 2: Preprocess input data
Transform raw input data (images, arrays) into MNN VARP tensors using MNN.cv for image operations and MNN.numpy for numerical operations. Apply model-specific normalization, resizing, and format conversion. Ensure the tensor's memory layout (NHWC from MNN.cv or NCHW from MNN.numpy) matches the model's expected input format.
Key considerations:
- MNN.cv produces NHWC-layout tensors; MNN.numpy produces NCHW-layout tensors
- Use MNN.expr.convert() to change between NC4HW4, NHWC, and NCHW formats
- Use var.set_order() to explicitly set the memory layout if needed
- Keep preprocessing logic minimal to avoid becoming a performance bottleneck
Step 3: Configure runtime and load model
Create a RuntimeManager with backend selection (CPU=0, Metal=1, CUDA=2, OpenCL=3, NPU=5, Vulkan=7), precision mode (normal=0, high=1, low=2), memory strategy, and thread count. Load the model using nn.load_module_from_file, specifying input and output tensor names. Optionally set shape_mutable=False for static input shapes and configure a GPU cache file for faster subsequent initializations.
What happens:
- RuntimeManager allocates backend resources and configures thread pool
- Model graph is loaded from the .mnn file and scheduled across the selected backend(s)
- For GPU backends, kernel compilation occurs on first run (cacheable for subsequent runs)
- Setting memory=low enables dynamic quantization for weight-quantized models
Step 4: Execute inference
Pass preprocessed VARP tensors to the model's forward method. The module accepts a list of input VARPs and returns a list of output VARPs. For GPU backends with static shapes, the first forward call may include initialization overhead.
Key considerations:
- Input list order must match the input_names specified during model loading
- Output list order corresponds to the output_names specified during model loading
- The forward method is reusable across multiple inference calls without reloading
Step 5: Post-process output
Extract results from output VARP tensors using MNN.numpy operations (argmax, reshape, etc.) or convert to Python types using read_as_tuple(). Apply application-specific post-processing such as class label lookup, bounding box decoding, or text generation.
Key considerations:
- Use var.shape to inspect output dimensions
- Use var.read_as_tuple() to convert tensor data to Python tuples
- MNN.numpy operations can be used for further numerical processing on outputs