Make-A-Video: Text-to-Video Generation without Text-Video Data

Abstract

We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.

Cited by in corpus (16)

• A Taxonomy of Prompt Modifiers for Text-To-Image Generation
• Break-A-Scene: Extracting Multiple Concepts from a Single Image
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• A Survey of AI Text-to-Image and AI Text-to-Video Generators
• ZenSVI: An Open-Source Software for the Integrated Acquisition, Processing and Analysis of Street View Imagery Towards Scalable Urban Science
• DALLE-URBAN: Capturing the urban design expertise of large text to image transformers
• Quantum circuit synthesis with diffusion models
• CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy
• Situating the social issues of image generation models in the model life cycle: a sociotechnical approach
• Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image
• LaMD: Latent Motion Diffusion for Image-Conditional Video Generation
• F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis
• Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
• Dreaming of Electrical Waves: Generative Modeling of Cardiac Excitation Waves using Diffusion Models
• A Multimodal Symphony: Integrating Taste and Sound through Generative AI
• Controllable 3D Object Generation with Single Image Prompt