Transformation-based Adversarial Video Prediction on Large-Scale Data
arXiv:2003.04035
Abstract
Recent breakthroughs in adversarial generative modeling have led to models capable of producing video samples of high quality, even on large and complex datasets of real-world video. In this work, we focus on the task of video prediction, where given a sequence of frames extracted from a video, the goal is to generate a plausible future sequence. We first improve the state of the art by performing a systematic empirical study of discriminator decompositions and proposing an architecture that yields faster convergence and higher performance than previous approaches. We then analyze recurrent units in the generator, and propose a novel recurrent unit which transforms its past hidden state according to predicted motion-like features, and refines it to handle dis-occlusions, scene changes and other complex behavior. We show that this recurrent unit consistently outperforms previous designs. Our final model leads to a leap in the state-of-the-art performance, obtaining a test set Frechet Video Distance of 25.7, down from 69.2, on the large-scale Kinetics-600 dataset.
References in corpus (35)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- The Kinetics Human Action Video Dataset
- Self-Attention Generative Adversarial Networks
- MesoNet: a Compact Facial Video Forgery Detection Network
- Unsupervised Learning of Video Representations using LSTMs
- A Learned Representation For Artistic Style
- MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Large Scale Adversarial Representation Learning
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- Recurrent Convolutional Strategies for Face Manipulation Detection in Videos
- Stochastic Video Generation with a Learned Prior
- A Short Note about Kinetics-600
- Modulating early visual processing by language
- Video (language) modeling: a baseline for generative models of natural videos
- Multimodal Unsupervised Image-to-Image Translation
- Exploration by Random Network Distillation
- Spatio-temporal video autoencoder with differentiable memory
- Stochastic Adversarial Video Prediction
- Towards Accurate Generative Models of Video: A New Metric & Challenges
- Delving Deeper into Convolutional Networks for Learning Video Representations
- Adversarial Video Generation on Complex Datasets
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
- Geometric GAN
- Transformation-Based Models of Video Sequences
- Object-Oriented Dynamics Predictor
- Deep Visual Foresight for Planning Robot Motion
- Video Prediction with Appearance and Motion Conditions