Imagen Video: High Definition Video Generation with Diffusion Models

Abstract

We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models. Given a text prompt, Imagen Video generates high definition videos using a base video generation model and a sequence of interleaved spatial and temporal video super-resolution models. We describe how we scale up the system as a high definition text-to-video model including design decisions such as the choice of fully-convolutional temporal and spatial super-resolution models at certain resolutions, and the choice of the v-parameterization of diffusion models. In addition, we confirm and transfer findings from previous work on diffusion-based image generation to the video generation setting. Finally, we apply progressive distillation to our video models with classifier-free guidance for fast, high quality sampling. We find Imagen Video not only capable of generating videos of high fidelity, but also having a high degree of controllability and world knowledge, including the ability to generate diverse videos and text animations in various artistic styles and with 3D object understanding. See https://imagen.research.google/video/ for samples.

arXiv commentSee accompanying website: https://imagen.research.google/video/

Cited by in corpus (21)

• Reproducible scaling laws for contrastive language-image learning
• Inverse-design of nonlinear mechanical metamaterials via video denoising diffusion models
• A Taxonomy of Prompt Modifiers for Text-To-Image Generation
• PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
• Diffusion Model-Based Image Editing: A Survey
• SADM: Sequence-Aware Diffusion Model for Longitudinal Medical Image Generation
• A Survey of AI Text-to-Image and AI Text-to-Video Generators
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• The Infinite Index: Information Retrieval on Generative Text-To-Image Models
• Drastic Circuit Depth Reductions with Preserved Adversarial Robustness by Approximate Encoding for Quantum Machine Learning
• ExpressEdit: Video Editing with Natural Language and Sketching
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• Investigating the Design Space of Diffusion Models for Speech Enhancement
• 4D Facial Expression Diffusion Model
• Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image
• Situating the social issues of image generation models in the model life cycle: a sociotechnical approach
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models
• F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
• Table-to-Text Generation with Pretrained Diffusion Models
• Infusion: internal diffusion for inpainting of dynamic textures and complex motion
• Diff-3DCap: Shape Captioning with Diffusion Models