Principle:Tencent Ncnn Video Matting

Knowledge Sources	Tencent_Ncnn
Domains	Computer Vision, Video Processing
Last Updated	2026-02-09 19:00 GMT

Overview

The prediction of per-pixel alpha matte values using recurrent hidden states that propagate temporal context across frames, enabling temporally consistent foreground separation in video sequences.

Description

Video matting extends image matting to the temporal domain, estimating a continuous alpha matte for every pixel in every frame of a video. The alpha matte describes the opacity of the foreground at each pixel, ranging from 0 (fully background) to 1 (fully foreground), with fractional values along boundaries capturing fine details like hair strands and semi-transparent regions.

The key challenge distinguishing video matting from per-frame image matting is temporal consistency. Applying an image matting model independently to each frame produces flickering artifacts because small perturbations in input (due to noise, compression, or subtle motion) cause disproportionate changes in the predicted matte. Video matting solves this by introducing recurrent hidden states that carry temporal context from one frame to the next.

In a recurrent video matting architecture, the model maintains an internal hidden state (typically a set of feature maps at multiple resolutions) that is updated at each frame and passed forward to the next. This hidden state acts as a temporal memory, encoding information about the foreground's appearance and position in recent frames. When processing the current frame, the model fuses the visual features of the current frame with the recurrent hidden state, allowing it to make predictions that are consistent with previous frames while still adapting to new content.

The recurrent design is preferable to approaches that require future frames (bidirectional models) because it supports real-time streaming inference, processing one frame at a time with constant memory usage regardless of video length.

Usage

This principle applies in video applications requiring foreground-background separation:

• Video conferencing: Replacing or blurring the background behind a person in real time.
• Film production: Extracting actors from green screens or natural backgrounds for compositing.
• Live streaming: Overlaying streamers onto virtual backgrounds or game footage.
• Augmented reality: Separating the user from the real background for mixed-reality effects.

Theoretical Basis

The recurrent matting model processes frames sequentially:

// Initialize recurrent states (typically as zeros)
r1, r2, r3, r4 = zeros(), zeros(), zeros(), zeros()

for each frame in video:
    // Downsample input for efficiency
    src_sm = downsample(frame, scale=0.25)

    // Extract visual features
    f1, f2, f3, f4 = Encoder(src_sm)

    // Fuse features with recurrent hidden states
    f4, r4 = RecurrentBlock(f4, r4)    // deepest resolution
    f3, r3 = RecurrentBlock(f3, r3)
    f2, r2 = RecurrentBlock(f2, r2)
    f1, r1 = RecurrentBlock(f1, r1)    // highest resolution

    // Decode alpha matte and foreground
    alpha_sm, fgr_sm = Decoder(f1, f2, f3, f4)

    // Upsample to original resolution with detail refinement
    alpha, fgr = Refiner(frame, alpha_sm, fgr_sm)

    // Composite with desired background
    output = fgr * alpha + new_background * (1 - alpha)

The alpha compositing equation for blending foreground and background:

$C_{out}=\alpha \cdot C_{fg}+(1-\alpha )\cdot C_{bg}$

where $\alpha \in [0,1]$ is the predicted matte value, $C_{fg}$ is the foreground color, and $C_{bg}$ is the replacement background color.

Each recurrent block updates its hidden state via a gated mechanism:

function RecurrentBlock(features, prev_state):
    combined = concatenate(features, prev_state)
    gate = sigmoid(Conv(combined))
    new_state = gate * Conv(combined) + (1 - gate) * prev_state
    output = ReLU(Conv(concatenate(features, new_state)))
    return output, new_state

Related Pages

• Implementation:Tencent_Ncnn_RVM_Example