FitVid: Overfitting in Pixel-Level Video Prediction

Abstract

An agent that is capable of predicting what happens next can perform a variety of tasks through planning with no additional training. Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for a variety of visual perception tasks. This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances. Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain. There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions. In this paper, we argue that the inefficient use of parameters in the current video models is the main reason for underfitting. Therefore, we introduce a new architecture, named FitVid, which is capable of severe overfitting on the common benchmarks while having similar parameter count as the current state-of-the-art models. We analyze the consequences of overfitting, illustrating how it can produce unexpected outcomes such as generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques. As a result, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.

References in corpus (39)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Gaussian Error Linear Units (GELUs)
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Unsupervised Learning of Video Representations using LSTMs
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Generating Videos with Scene Dynamics
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Searching for Activation Functions
• Density estimation using Real NVP
• A Recurrent Latent Variable Model for Sequential Data
• DeepFakes: a New Threat to Face Recognition? Assessment and Detection
• NVAE: A Deep Hierarchical Variational Autoencoder
• Learning Latent Dynamics for Planning from Pixels
• Stochastic Video Generation with a Learned Prior
• A Review on Deep Learning Techniques for Video Prediction
• Video (language) modeling: a baseline for generative models of natural videos
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Stochastic Adversarial Video Prediction
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Early Visual Concept Learning with Unsupervised Deep Learning
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• Stochastic Latent Residual Video Prediction
• Transformation-Based Models of Video Sequences
• Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
• Predicting Video with VQVAE
• Unsupervised Learning of Object Structure and Dynamics from Videos
• Offline Reinforcement Learning from Images with Latent Space Models
• Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• On the difficulty of learning and predicting the long-term dynamics of bouncing objects