Flexible Neural Representation for Physics Prediction

Abstract

Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail. Inspired by this ability, we propose a hierarchical particle-based object representation that covers a wide variety of types of three-dimensional objects, including both arbitrary rigid geometrical shapes and deformable materials. We then describe the Hierarchical Relation Network (HRN), an end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation. Compared to other neural network baselines, the HRN accurately handles complex collisions and nonrigid deformations, generating plausible dynamics predictions at long time scales in novel settings, and scaling to large scene configurations. These results demonstrate an architecture with the potential to form the basis of next-generation physics predictors for use in computer vision, robotics, and quantitative cognitive science.

23 pages, 20 figures

References in corpus (18)

• Semi-Supervised Classification with Graph Convolutional Networks
• Geometric deep learning: going beyond Euclidean data
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Convolutional Networks on Graph-Structured Data
• Deep Convolutional Inverse Graphics Network
• Gated Graph Sequence Neural Networks
• Unsupervised Learning for Physical Interaction through Video Prediction
• Neural Relational Inference for Interacting Systems
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Learning Visual Predictive Models of Physics for Playing Billiards
• Modeling Relational Data with Graph Convolutional Networks
• Visual Interaction Networks
• To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
• Understanding Visual Concepts with Continuation Learning
• IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
• Probing Physics Knowledge Using Tools from Developmental Psychology
• Inverse Graphics with Probabilistic CAD Models
• 3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations

Cited by in corpus (35)

• Learning to Simulate Complex Physics with Graph Networks
• Discovering Symbolic Models from Deep Learning with Inductive Biases
• Object-Centric Learning with Slot Attention
• Towards Sparse Hierarchical Graph Classifiers
• Multipole Graph Neural Operator for Parametric Partial Differential Equations
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• What Can Neural Networks Reason About?
• Teaching the Incompressible Navier-Stokes Equations to Fast Neural Surrogate Models in 3D
• Learning Compositional Koopman Operators for Model-Based Control
• Causal Discovery in Physical Systems from Videos
• Pragmatic inference and visual abstraction enable contextual flexibility during visual communication
• Interactive Differentiable Simulation
• Manipulation of Granular Materials by Learning Particle Interactions
• Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact
• HighAir: A Hierarchical Graph Neural Network-Based Air Quality Forecasting Method
• Visual Grounding of Learned Physical Models
• Graph Element Networks: adaptive, structured computation and memory
• ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics
• Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Learning Physical Constraints with Neural Projections
• Unsupervised Discovery of 3D Physical Objects from Video
• Forward Prediction for Physical Reasoning
• Physics Encoded Blocks in Residual Neural Network Architectures for Digital Twin Models
• Learning Incompressible Fluid Dynamics from Scratch -- Towards Fast, Differentiable Fluid Models that Generalize
• Vid2Param: Modelling of Dynamics Parameters from Video
• Accurately Solving Physical Systems with Graph Learning
• Deep learning reveals hidden interactions in complex systems
• Integrating Physics and Topology in Neural Networks for Learning Rigid Body Dynamics
• Modular meta-learning in abstract graph networks for combinatorial generalization
• Generative Deep Learning Model for a Multi-level Nano-Optic Broadband Power Splitter
• Sparse hierarchical representation learning on molecular graphs
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Dynamic Relational Inference in Multi-Agent Trajectories