A Compositional Object-Based Approach to Learning Physical Dynamics

Abstract

We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.

Published as a conference paper for ICLR 2017. 15 pages, 6 figures

Cited by in corpus (78)

• Graph Neural Networks: A Review of Methods and Applications
• Explainable Machine Learning for Scientific Insights and Discoveries
• Discovering physical concepts with neural networks
• Few-Shot Learning with Graph Neural Networks
• Neural Relational Inference for Interacting Systems
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Learning to Decompose and Disentangle Representations for Video Prediction
• What Can Neural Networks Reason About?
• Hamiltonian Graph Networks with ODE Integrators
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• Symplectic Recurrent Neural Networks
• Explainability Techniques for Graph Convolutional Networks
• On the Binding Problem in Artificial Neural Networks
• Learning Compositional Koopman Operators for Model-Based Control
• Differentiable Physics-informed Graph Networks
• gradSim: Differentiable simulation for system identification and visuomotor control
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• Learning A Physical Long-term Predictor
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• CoPhy: Counterfactual Learning of Physical Dynamics
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations
• Learning Robotic Manipulation of Granular Media
• Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact
• Black-Box Optimization with Local Generative Surrogates
• Differentiable solver for time-dependent deformation problems with contact
• Visual Grounding of Learned Physical Models
• Multi-agent Trajectory Prediction with Fuzzy Query Attention
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
• ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators
• Active Learning of Abstract Plan Feasibility
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
• Propagation Networks for Model-Based Control Under Partial Observation
• Learning Physical Constraints with Neural Projections
• Detecting Beneficial Feature Interactions for Recommender Systems
• Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
• Forward Prediction for Physical Reasoning
• Occlusion resistant learning of intuitive physics from videos
• Object-centric Forward Modeling for Model Predictive Control
• Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models
• Perceiving Physical Equation by Observing Visual Scenarios
• GN-Transformer: Fusing Sequence and Graph Representation for Improved Code Summarization
• A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories
• Vid2Param: Modelling of Dynamics Parameters from Video
• Active World Model Learning with Progress Curiosity
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• Graph Neural Processes: Towards Bayesian Graph Neural Networks
• Programmable Agents
• Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videos
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Learning Generalizable Physical Dynamics of 3D Rigid Objects
• Unsupervised Intuitive Physics from Past Experiences
• Efficient Cloth Simulation using Miniature Cloth and Upscaling Deep Neural Networks
• ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking
• Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks
• RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks
• Relational State-Space Model for Stochastic Multi-Object Systems
• Structured agents for physical construction
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• On the difficulty of learning and predicting the long-term dynamics of bouncing objects
• Learning Representation over Dynamic Graph using Aggregation-Diffusion Mechanism
• Neural Consciousness Flow
• Interpretable Intuitive Physics Model
• End-to-End Learning from Complex Multigraphs with Latent-Graph Convolutional Networks
• Predicting the Physical Dynamics of Unseen 3D Objects
• Adding Intuitive Physics to Neural-Symbolic Capsules Using Interaction Networks
• MagNet: Discovering Multi-agent Interaction Dynamics using Neural Network
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Compositional Video Prediction
• Pushing it out of the Way: Interactive Visual Navigation