Workflow:Huggingface Diffusers Video Generation
| Knowledge Sources | |
|---|---|
| Domains | Diffusion_Models, Video_Generation, Inference |
| Last Updated | 2026-02-13 21:00 GMT |
Overview
End-to-end process for generating video sequences from text prompts or reference images using video diffusion pipelines.
Description
This workflow covers generating video content using diffusion models that operate on spatiotemporal latent representations. Video diffusion models extend image generation to the temporal dimension, producing sequences of coherent frames. The workflow supports multiple generation modes: text-to-video (generating a video from a text description), image-to-video (animating a reference image into a video sequence), and video-to-video (transforming an existing video while following a text prompt). Supported model families include Wan (v1, v2.2, VACE, Animate), HunyuanVideo (v1, v1.5, FramePack), CogVideoX, SkyReels V2, LTX/LTX2, Sana Video, Stable Video Diffusion, AnimateDiff, Mochi, Allegro, and Latte. The workflow handles the unique memory and computational challenges of video generation, including temporal attention, 3D VAE encoding, and frame-by-frame export.
Usage
Execute this workflow when you need to generate video content from text descriptions, animate a still image into a video clip, or restyle an existing video. This workflow is appropriate for creative content generation, prototype animation, visual effects previews, and any scenario where automated video production is needed. Video generation requires significantly more memory and computation than image generation, so memory optimization strategies are essential.
Execution Steps
Step 1: Pipeline Selection
Select the appropriate video generation pipeline based on the desired generation mode (text-to-video, image-to-video, or video-to-video) and the target model family. Each model family offers different tradeoffs in video quality, temporal coherence, generation speed, and memory requirements.
Key considerations:
- Wan v2.2 and HunyuanVideo are among the highest-quality open video models
- CogVideoX supports both text-to-video and image-to-video modes
- AnimateDiff extends Stable Diffusion with motion modules for animation
- Stable Video Diffusion (SVD) specializes in image-to-video animation
- FramePack (HunyuanVideo 1.5) enables progressive video generation
- Model selection depends on desired video length, resolution, and quality
Step 2: Model Loading and Memory Setup
Load the video diffusion pipeline with aggressive memory optimizations. Video models are substantially larger than image models due to temporal attention layers and 3D VAE components. Configure offloading strategies to manage the large memory footprint across multiple forward passes.
Key considerations:
- Video models often require 24GB+ VRAM without optimization
- Model CPU offloading is essential for consumer GPUs
- Group offloading (via hooks) provides finer-grained memory management
- VAE tiling is critical for decoding video latents within memory limits
- Enable VAE slicing to process frames sequentially during encoding/decoding
- Some models support attention processor swapping for memory savings
Step 3: Input Preparation
Prepare the input conditioning based on the generation mode. For text-to-video, craft a descriptive prompt. For image-to-video, load and preprocess the reference image to the model's expected resolution. For video-to-video, load and encode the source video frames.
Key considerations:
- Video prompts benefit from describing motion and temporal progression
- Reference images for image-to-video should be high quality and at the target resolution
- Some models accept negative prompts to avoid undesired artifacts
- WAN VACE supports combined conditioning (image + video + mask) for versatile control
- Frame count, FPS, and resolution are configured at this stage
Step 4: Video Denoising
Execute the spatiotemporal denoising loop. The model processes a 3D latent tensor (frames x height x width) through iterative noise removal. Temporal attention layers ensure frame-to-frame coherence while spatial attention generates per-frame detail. The process is analogous to image denoising but operates across the temporal dimension simultaneously.
Key considerations:
- Video generation typically requires 30-50 denoising steps
- Guidance scale controls prompt adherence (typically 5.0-9.0 for video models)
- Higher frame counts increase memory requirements and generation time linearly
- Some models support progressive generation (FramePack) for arbitrarily long videos
- Flow-matching schedulers are used by newer models (Wan, HunyuanVideo)
Step 5: Video Decoding and Export
Decode the denoised 3D latent tensor into pixel-space video frames using the 3D VAE decoder, then export the frames as a video file. The VAE processes the temporal dimension alongside spatial dimensions to ensure smooth inter-frame transitions.
Key considerations:
- 3D VAE decoding is memory-intensive; use tiling for large resolutions
- Export formats include MP4 (via imageio/ffmpeg), GIF, or individual frame PNGs
- Frame rate (FPS) should match the model's training distribution (typically 8-24 FPS)
- Post-processing may include frame interpolation for smoother playback
- The export_to_video utility function handles the common MP4 export path