Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
arXiv:2008.02265
Abstract
Learning long-term dynamics models is the key to understanding physical common sense. Most existing approaches on learning dynamics from visual input sidestep long-term predictions by resorting to rapid re-planning with short-term models. This not only requires such models to be super accurate but also limits them only to tasks where an agent can continuously obtain feedback and take action at each step until completion. In this paper, we aim to leverage the ideas from success stories in visual recognition tasks to build object representations that can capture inter-object and object-environment interactions over a long-range. To this end, we propose Region Proposal Interaction Networks (RPIN), which reason about each object's trajectory in a latent region-proposal feature space. Thanks to the simple yet effective object representation, our approach outperforms prior methods by a significant margin both in terms of prediction quality and their ability to plan for downstream tasks, and also generalize well to novel environments. Code, pre-trained models, and more visualization results are available at https://haozhi.io/RPIN.
arXiv commentICLR 2021; Code: https://github.com/HaozhiQi/RPIN Website: https://haozhiqi.github.io/RPIN/ v5: update PHYRE results of each evaluation fold
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Interaction Networks for Learning about Objects, Relations and Physics
- Learning to Simulate Complex Physics with Graph Networks
- Decomposing Motion and Content for Natural Video Sequence Prediction
- Deep multi-scale video prediction beyond mean square error
- Stochastic Video Generation with a Learned Prior
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Stochastic Adversarial Video Prediction
- Learning to Generate Long-term Future via Hierarchical Prediction
- A Compositional Object-Based Approach to Learning Physical Dynamics
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Learning Visual Predictive Models of Physics for Playing Billiards
- Learning Physical Intuition of Block Towers by Example
- Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
- Learning A Physical Long-term Predictor
- Video Imagination from a Single Image with Transformation Generation
- Structured Object-Aware Physics Prediction for Video Modeling and Planning
- Forward Prediction for Physical Reasoning