GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions

Abstract

Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.

References in corpus (16)

• Learning Transferable Visual Models From Natural Language Supervision
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• The Kinetics Human Action Video Dataset
• Neural Discrete Representation Learning
• Self-Attention Generative Adversarial Networks
• Zero-Shot Text-to-Image Generation
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Generating Videos with Scene Dynamics
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Stochastic Video Generation with a Learned Prior
• Unsupervised Learning for Physical Interaction through Video Prediction
• Neural Paraphrase Generation with Stacked Residual LSTM Networks
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Stochastic Variational Video Prediction
• Predicting Video with VQVAE

Cited by in corpus (8)

• Transflower: probabilistic autoregressive dance generation with multimodal attention
• A Survey of AI Text-to-Image and AI Text-to-Video Generators
• FitVid: Overfitting in Pixel-Level Video Prediction
• Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
• FastCLIPstyler: Optimisation-free Text-based Image Style Transfer Using Style Representations
• Language-Driven Image Style Transfer
• NWT: Towards natural audio-to-video generation with representation learning
• Don't Forget your Inverse DDIM for Image Editing