Predicting Video with VQVAE

Abstract

In recent years, the task of video prediction-forecasting future video given past video frames-has attracted attention in the research community. In this paper we propose a novel approach to this problem with Vector Quantized Variational AutoEncoders (VQ-VAE). With VQ-VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast to previous work that has largely emphasized highly constrained datasets, we focus on very diverse, large-scale datasets such as Kinetics-600. We predict video at a higher resolution on unconstrained videos, 256x256, than any other previous method to our knowledge. We further validate our approach against prior work via a crowdsourced human evaluation.

13 Pages

References in corpus (23)

• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Discrete Representation Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• MesoNet: a Compact Facial Video Forgery Detection Network
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Generating Videos with Scene Dynamics
• Dynamic Filter Networks
• Stochastic Video Generation with a Learned Prior
• A Short Note about Kinetics-600
• Video (language) modeling: a baseline for generative models of natural videos
• Spatio-temporal video autoencoder with differentiable memory
• Stochastic Adversarial Video Prediction
• Learning to Generate Long-term Future via Hierarchical Prediction
• Adversarial Video Generation on Complex Datasets
• The challenge of realistic music generation: modelling raw audio at scale
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Jukebox: A Generative Model for Music
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Predicting Deeper into the Future of Semantic Segmentation
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• Hierarchical Autoregressive Image Models with Auxiliary Decoders
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Flow-Grounded Spatial-Temporal Video Prediction from Still Images

Cited by in corpus (8)

• VideoGPT: Video Generation using VQ-VAE and Transformers
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• FitVid: Overfitting in Pixel-Level Video Prediction
• CHeart: A Conditional Spatio-Temporal Generative Model for Cardiac Anatomy
• SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches
• Unsupervised dynamic modeling of medical image transformation
• NWT: Towards natural audio-to-video generation with representation learning