Physion: Evaluating Physical Prediction from Vision in Humans and Machines

Abstract

While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.

arXiv comment28 pages

References in corpus (24)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Shortcut Learning in Deep Neural Networks
• Interaction Networks for Learning about Objects, Relations and Physics
• Stochastic Video Generation with a Learned Prior
• Stochastic Adversarial Video Prediction
• MONet: Unsupervised Scene Decomposition and Representation
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Flexible Neural Representation for Physics Prediction
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Learning Physical Intuition of Block Towers by Example
• COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
• Modeling human intuitions about liquid flow with particle-based simulation
• To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
• Relational Forward Models for Multi-Agent Learning
• Learning Physical Graph Representations from Visual Scenes
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
• IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
• Goal-Aware Prediction: Learning to Model What Matters
• Probing Physics Knowledge Using Tools from Developmental Psychology
• Forward Prediction for Physical Reasoning
• Occlusion resistant learning of intuitive physics from videos

Cited by in corpus (1)

• Integrating Physics and Topology in Neural Networks for Learning Rigid Body Dynamics