VideoGPT: Video Generation using VQ-VAE and Transformers

Abstract

We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html

arXiv commentProject website: https://wilson1yan.github.io/videogpt/index.html

References in corpus (36)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Denoising Diffusion Probabilistic Models
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• Unsupervised Learning of Video Representations using LSTMs
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• NICE: Non-linear Independent Components Estimation
• Zero-Shot Text-to-Image Generation
• Generative Modeling by Estimating Gradients of the Data Distribution
• Density estimation using Real NVP
• Generating Long Sequences with Sparse Transformers
• NVAE: A Deep Hierarchical Variational Autoencoder
• Axial Attention in Multidimensional Transformers
• MetNet: A Neural Weather Model for Precipitation Forecasting
• Stochastic Adversarial Video Prediction
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Non-local Neural Networks
• Adversarial Video Generation on Complex Datasets
• Video-to-Video Synthesis
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Self-Supervised Visual Planning with Temporal Skip Connections
• Jukebox: A Generative Model for Music
• High Fidelity Speech Synthesis with Adversarial Networks
• Stochastic Variational Video Prediction
• PixelSNAIL: An Improved Autoregressive Generative Model
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
• Lower Dimensional Kernels for Video Discriminators
• Predicting Video with VQVAE
• Scaling Autoregressive Video Models
• Transformation-based Adversarial Video Prediction on Large-Scale Data

Cited by in corpus (16)

• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• RockGPT: Reconstructing three-dimensional digital rocks from single two-dimensional slice from the perspective of video generation
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• FitVid: Overfitting in Pixel-Level Video Prediction
• CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy
• Taming Visually Guided Sound Generation
• Continuous PDE Dynamics Forecasting with Implicit Neural Representations
• Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• A Two-part Transformer Network for Controllable Motion Synthesis
• Recursive Variational Autoencoders for 3D Blood Vessel Generative Modeling
• NWT: Towards natural audio-to-video generation with representation learning
• VidStyleODE: Disentangled Video Editing via StyleGAN and NeuralODEs
• Layered Controllable Video Generation
• Pathologies in priors and inference for Bayesian transformers