Principle:Alibaba MNN Diffusion Inference Execution
| Field | Value |
|---|---|
| principle_name | Diffusion_Inference_Execution |
| schema_version | 0.3.0 |
| principle_type | Workflow Step |
| domain | Stable Diffusion Deployment |
| stage | Inference Execution |
| scope | Running text-to-image generation through iterative latent diffusion denoising |
| last_updated | 2026-02-10 14:00 GMT |
Overview
Diffusion Inference Execution is the fifth and final step in the Stable Diffusion deployment workflow. This step takes the compiled MNN engine and the converted MNN model files, and executes the complete text-to-image generation pipeline. The output is a generated image corresponding to the user's text prompt.
Theory
Stable Diffusion is a latent diffusion model (LDM). Unlike pixel-space diffusion models that operate directly on full-resolution images, LDMs perform the diffusion process in a compressed latent space, dramatically reducing computational cost while preserving image quality.
The inference pipeline consists of four sequential stages:
Stage 1: Text Encoding
The user's text prompt is first tokenized into a sequence of integer token IDs using the model's vocabulary (CLIP tokenizer for English models, bilingual CLIP tokenizer for Taiyi Chinese models). The token sequence is padded or truncated to a fixed length (typically 77 tokens). The text encoder (CLIP ViT-L/14) then maps these token IDs to a sequence of high-dimensional embedding vectors (the "hidden states"). These embeddings encode the semantic meaning of the prompt and serve as the conditioning signal for the denoising process.
Stage 2: Latent Noise Initialization
A random noise tensor is sampled from a standard Gaussian distribution in the latent space. The spatial dimensions of this tensor correspond to the target image resolution divided by the VAE's downsampling factor (typically 8x, so a 512x512 image uses a 64x64 latent). The random seed parameter controls the initial noise sample: the same seed with the same prompt produces an identical image, enabling reproducibility. A seed value of -1 indicates that a truly random seed should be used.
Stage 3: Iterative UNet Denoising
This is the core of the diffusion process. Starting from the random noise, the UNet iteratively predicts and removes noise over a series of timesteps, guided by the text encoder's hidden states:
- At each timestep t, the UNet receives three inputs: the current noisy latent, the timestep value, and the text encoder hidden states.
- The UNet predicts the noise component present in the current latent.
- A noise scheduler (e.g., PNDM, DDIM, or Euler) uses this prediction to compute the denoised latent for the next timestep.
- This process repeats for the specified number of iterations (typically 10-20 steps). More iterations generally produce higher-quality images but take proportionally longer.
The Classifier-Free Guidance (CFG) technique is commonly used: the UNet is run twice per step (once with the text conditioning, once with an unconditional/empty embedding), and the results are blended to strengthen adherence to the prompt.
Stage 4: VAE Decoding
After the final denoising step, the clean latent representation is passed through the VAE decoder, which upsamples and transforms it back into pixel-space RGB values. The resulting image is then written to disk as a JPEG or PNG file.
Memory Management Modes
The MNN diffusion engine supports three memory management strategies to accommodate different hardware constraints:
- Mode 0 (Memory Saving): Each sub-model (text encoder, UNet, VAE decoder) is loaded into memory only when needed and freed immediately after use. This minimizes peak memory usage (suitable for devices with 2 GB+ RAM) but incurs repeated model loading overhead.
- Mode 1 (Memory Enough): All sub-models are loaded into memory at initialization and remain resident. This provides the fastest generation speed but requires sufficient memory to hold all models simultaneously.
- Mode 2 (Balance): A balanced approach that keeps frequently-used models resident while swapping less-used components. This trades some speed for reduced memory footprint.
Backend Types
The MNN engine supports multiple hardware backends for acceleration:
- 0 (CPU): Default backend, works on all platforms
- 3 (OpenCL): GPU acceleration via OpenCL (Linux, Android)
- 4 (Metal): GPU acceleration via Apple Metal (macOS, iOS)