Forward Prediction for Physical Reasoning
arXiv:2006.10734
Abstract
Physical reasoning requires forward prediction: the ability to forecast what will happen next given some initial world state. We study the performance of state-of-the-art forward-prediction models in the complex physical-reasoning tasks of the PHYRE benchmark. We do so by incorporating models that operate on object or pixel-based representations of the world into simple physical-reasoning agents. We find that forward-prediction models can improve physical-reasoning performance, particularly on complex tasks that involve many objects. However, we also find that these improvements are contingent on the test tasks being small variations of train tasks, and that generalization to completely new task templates is challenging. Surprisingly, we observe that forward predictors with better pixel accuracy do not necessarily lead to better physical-reasoning performance.Nevertheless, our best models set a new state-of-the-art on the PHYRE benchmark.
arXiv commentWebpage/code/models: https://facebookresearch.github.io/phyre-fwd/
References in corpus (27)
- Adam: A Method for Stochastic Optimization
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Interaction Networks for Learning about Objects, Relations and Physics
- DeepMind Control Suite
- Recurrent World Models Facilitate Policy Evolution
- Hamiltonian Neural Networks
- Learning to Simulate Complex Physics with Graph Networks
- Learning Latent Dynamics for Planning from Pixels
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- Learning to Generate Long-term Future via Hierarchical Prediction
- Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
- A Compositional Object-Based Approach to Learning Physical Dynamics
- Flexible Neural Representation for Physics Prediction
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
- Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
- Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
- Learning Physical Intuition of Block Towers by Example
- Symplectic Recurrent Neural Networks
- Learning and Querying Fast Generative Models for Reinforcement Learning
- To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
- Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
- IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
- Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
- 3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations
- ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking