Implementation:Tencent Ncnn RVM Example
| Knowledge Sources | |
|---|---|
| Domains | Vision, Video_Matting |
| Last Updated | 2026-02-09 19:00 GMT |
Overview
Concrete tool for background removal and video matting inference using Robust Video Matting (RVM) with ncnn.
Description
This example implements Robust Video Matting using a MobileNetV3-based matting network exported via PNNX. The model processes images with recurrent state tensors (r1-r4) that persist across frames for temporal consistency in video applications. For MobileNetV3, the recurrent states have channel sizes of 16, 20, 40, and 64 at progressively downsampled spatial resolutions.
When the input image exceeds 512 pixels on the longer side, the model operates in a downsampled mode: the full-resolution image is padded and used as the high-res input (in0), while a resized version serves as the low-res input (in1). The model supports three output modes: (1) basic downsampled matting (out0/out1), (2) deep guided filter refinement (out2/out3) for higher quality, and (3) fast guided filter refinement (out4/out5) for speed. It also produces a segmentation mask (out6) and updated recurrent states (out7-out10).
The input is normalized to [0,1] range (divided by 255). The output foreground (fgr), alpha matte (pha), and segmentation mask (seg) are denormalized back to [0,255] and converted to OpenCV Mats. The composite visualization blends foreground with a green background using the alpha matte. Letterbox padding is applied and removed to handle arbitrary input sizes aligned to a stride of 16.
Usage
Use this example to remove image backgrounds, generate alpha mattes, or produce segmentation masks. Designed for video matting with temporal consistency via recurrent states, but also works on single images.
Code Reference
Source Location
- Repository: Tencent_Ncnn
- File: examples/rvm.cpp
- Lines: 1-336
Signature
static int detect_rvm(const cv::Mat& bgr, cv::Mat& fgr, cv::Mat& pha, cv::Mat& seg);
static void draw_objects(const cv::Mat& bgr, const cv::Mat& fgr, const cv::Mat& pha, const cv::Mat& seg);
int main(int argc, char** argv);
Import
#include "net.h"
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| imagepath | const char* (argv[1]) | Yes | Path to input image |
Outputs
| Name | Type | Description |
|---|---|---|
| fgr | cv::Mat (CV_8UC3) | Foreground image with background removed |
| pha | cv::Mat (CV_8UC1) | Alpha matte (0=background, 255=foreground) |
| seg | cv::Mat (CV_8UC1) | Segmentation mask |
| visualization | cv::Mat | Composite image (foreground over green) and segmented overlay displayed via cv::imshow |
Usage Examples
Running the Example
./rvm image.jpg
Key Code Pattern
ncnn::Net rvm;
rvm.opt.use_vulkan_compute = true;
rvm.load_param("rvm_mobilenetv3.ncnn.param");
rvm.load_model("rvm_mobilenetv3.ncnn.bin");
const float norm_vals[3] = {1 / 255.f, 1 / 255.f, 1 / 255.f};
ncnn::Mat in_pad, in_small_pad;
// ... letterbox resize and pad to stride 16
in_pad.substract_mean_normalize(0, norm_vals);
in_small_pad.substract_mean_normalize(0, norm_vals);
// Initialize recurrent states (zero-filled for first frame)
ncnn::Mat r1(in_small_pad.w / 2, in_small_pad.h / 2, 16);
ncnn::Mat r2(in_small_pad.w / 4, in_small_pad.h / 4, 20);
ncnn::Mat r3(in_small_pad.w / 8, in_small_pad.h / 8, 40);
ncnn::Mat r4(in_small_pad.w / 16, in_small_pad.h / 16, 64);
r1.fill(0.f); r2.fill(0.f); r3.fill(0.f); r4.fill(0.f);
ncnn::Extractor ex = rvm.create_extractor();
ex.input("in0", in_pad);
ex.input("in1", in_small_pad);
ex.input("in2", r1); ex.input("in3", r2);
ex.input("in4", r3); ex.input("in5", r4);
ncnn::Mat out_fgr, out_pha;
ex.extract("out2", out_fgr); // deep refined foreground
ex.extract("out3", out_pha); // deep refined alpha
ex.extract("out6", out_seg); // segmentation mask
// Updated recurrent states: out7-out10