On the difficulty of learning and predicting the long-term dynamics of bouncing objects
arXiv:1907.13494
Abstract
The ability to accurately predict the surrounding environment is a foundational principle of intelligence in biological and artificial agents. In recent years, a variety of approaches have been proposed for learning to predict the physical dynamics of objects interacting in a visual scene. Here we conduct a systematic empirical evaluation of several state-of-the-art unsupervised deep learning models that are considered capable of learning the spatio-temporal structure of a popular dataset composed by synthetic videos of bouncing objects. We show that most of the models indeed obtain high accuracy on the standard benchmark of predicting the next frame of a sequence, and one of them even achieves state-of-the-art performance. However, all models fall short when probed with the more challenging task of generating multiple successive frames. Our results show that the ability to perform short-term predictions does not imply that the model has captured the underlying structure and dynamics of the visual environment, thereby calling for a careful rethinking of the metrics commonly adopted for evaluating temporal models. We also investigate whether the learning outcome could be affected by the use of curriculum-based teaching.
References in corpus (12)
- Sequence to Sequence Learning with Neural Networks
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Representation Learning with Contrastive Predictive Coding
- Learning deep representations by mutual information estimation and maximization
- Unsupervised Learning of Video Representations using LSTMs
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Deep multi-scale video prediction beyond mean square error
- Learning to Execute
- Video (language) modeling: a baseline for generative models of natural videos
- A Compositional Object-Based Approach to Learning Physical Dynamics
- Unsupervised Learning of Visual Structure using Predictive Generative Networks
- Deep Temporal Sigmoid Belief Networks for Sequence Modeling