Principle:Zai org CogVideo 3D Video Decoding
| Knowledge Sources | |
|---|---|
| Domains | Video_Generation, Autoencoding, 3D_Decoding |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
3D video decoding reconstructs video sequences from quantized latent representations using hierarchical multi-resolution upsampling with causal temporal convolutions and quantized-code-conditioned spatial normalization.
Description
3D video decoding is an architectural principle for reconstructing video from compressed latent representations that have been discretized through vector quantization. Unlike simple 2D decoders applied frame-by-frame, 3D video decoders process the full spatio-temporal volume jointly, using causal 3D convolutions that respect temporal ordering (each output frame only depends on current and past frames, not future frames).
A key innovation in this approach is quantized-code-conditioned normalization, inspired by SPADE (Spatially-Adaptive Normalization). At each normalization layer, the original quantized codes are used to modulate feature maps through learned affine transforms. This allows the decoder to leverage the discrete codebook information at every stage of reconstruction, not just at the input, improving reconstruction fidelity.
The decoder operates in a multi-resolution hierarchy, progressively upsampling both spatial and temporal dimensions. Temporal upsampling is controlled independently from spatial upsampling, allowing different compression ratios along the spatial and temporal axes. Higher resolution levels may upsample only spatially while lower levels upsample both spatially and temporally.
Usage
Apply this principle when designing decoders for video VQ-VAE systems where temporal coherence and reconstruction quality are critical. It is particularly suited for systems where the latent space is significantly compressed along the temporal dimension and where the quantized codes carry structural information that should inform all stages of decoding.
Theoretical Basis
Causal Temporal Convolutions
Causal convolutions ensure that the output at time t depends only on inputs at times <= t. For a 3D convolution with temporal kernel size k_t:
Standard padding: pad both sides with (k_t - 1) / 2
Causal padding: pad only the left (past) side with (k_t - 1), zero on right
y(t) = sum_{tau=0}^{k_t-1} w(tau) * x(t - tau)
This preserves the autoregressive property in the temporal dimension while allowing free spatial processing.
Quantized-Code-Conditioned Normalization
Inspired by SPADE, feature maps are modulated by the quantized codes at each normalization layer:
Given features f of shape (B, C, T, H, W) and quantized codes zq of shape (B, C_zq, T', H', W'): 1. Interpolate zq to match f's spatial-temporal dimensions: zq_resized = interpolate(zq, size=(T, H, W)) 2. Optionally refine with a convolution: zq_resized = Conv3D(zq_resized) [if add_conv is enabled] 3. Compute learned scale and bias: gamma = Conv_1x1(zq_resized) [shape: (B, C, T, H, W)] beta = Conv_1x1(zq_resized) [shape: (B, C, T, H, W)] 4. Apply affine normalization: f_norm = GroupNorm(f) f_out = gamma * f_norm + beta
The first frame is handled separately from remaining frames during interpolation to maintain causal consistency.
Multi-Resolution Temporal Upsampling
The decoder uses a configurable number of temporal upsampling levels determined by the temporal compression factor:
temporal_compress_level = log2(temporal_compress_times)
For each resolution level i (from coarsest to finest):
if i >= num_resolutions - temporal_compress_level:
upsample spatially AND temporally (2x each)
else:
upsample spatially only (2x spatial, 1x temporal)
This allows, for example, 4x temporal compression with 2 levels of temporal upsampling and additional levels of spatial-only upsampling.
Decoder Architecture Flow
The complete decoding pipeline follows this structure:
Input: z of shape (B, C_z, T_compressed, H_compressed, W_compressed)
1. conv_in: CausalConv3d(z_channels -> block_in)
2. mid_block_1: ResnetBlock3D with SpatialNorm3D(zq)
3. mid_block_2: ResnetBlock3D with SpatialNorm3D(zq)
4. For each resolution level (coarse to fine):
For each residual block:
ResnetBlock3D with SpatialNorm3D(zq)
[optional] AttnBlock2D with SpatialNorm3D(zq)
[optional] Upsample3D (spatial and/or temporal)
5. norm_out: SpatialNorm3D(zq)
6. nonlinearity (swish)
7. conv_out: CausalConv3d(block_in -> out_ch)