Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction
arXiv:2103.04174
Abstract
A video prediction model that generalizes to diverse scenes would enable intelligent agents such as robots to perform a variety of tasks via planning with the model. However, while existing video prediction models have produced promising results on small datasets, they suffer from severe underfitting when trained on large and diverse datasets. To address this underfitting challenge, we first observe that the ability to train larger video prediction models is often bottlenecked by the memory constraints of GPUs or TPUs. In parallel, deep hierarchical latent variable models can produce higher quality predictions by capturing the multi-level stochasticity of future observations, but end-to-end optimization of such models is notably difficult. Our key insight is that greedy and modular optimization of hierarchical autoencoders can simultaneously address both the memory constraints and the optimization challenges of large-scale video prediction. We introduce Greedy Hierarchical Variational Autoencoders (GHVAEs), a method that learns high-fidelity video predictions by greedily training each level of a hierarchical autoencoder. In comparison to state-of-the-art models, GHVAEs provide 17-55% gains in prediction performance on four video datasets, a 35-40% higher success rate on real robot tasks, and can improve performance monotonically by simply adding more modules.
arXiv commentEqual advising and contribution for last two authors
References in corpus (25)
- Denoising Diffusion Probabilistic Models
- Language Models are Few-Shot Learners
- Conditional Image Generation with PixelCNN Decoders
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- NVAE: A Deep Hierarchical Variational Autoencoder
- Stochastic Video Generation with a Learned Prior
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Multi-Object Representation Learning with Iterative Variational Inference
- Stochastic Adversarial Video Prediction
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Towards Deeper Understanding of Variational Autoencoding Models
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Denoising Diffusion Implicit Models
- Hierarchical Long-term Video Prediction without Supervision
- Putting An End to End-to-End: Gradient-Isolated Learning of Representations
- Stochastic Latent Residual Video Prediction
- Latent Space Policies for Hierarchical Reinforcement Learning
- COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
- Transformation-Based Models of Video Sequences
- Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
- Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
- Goal-Aware Prediction: Learning to Model What Matters
- Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight