Interaction Networks for Learning about Objects, Relations and Physics

Abstract

Reasoning about objects, relations, and physics is central to human intelligence, and a key goal of artificial intelligence. Here we introduce the interaction network, a model which can reason about how objects in complex systems interact, supporting dynamical predictions, as well as inferences about the abstract properties of the system. Our model takes graphs as input, performs object- and relation-centric reasoning in a way that is analogous to a simulation, and is implemented using deep neural networks. We evaluate its ability to reason about several challenging physical domains: n-body problems, rigid-body collision, and non-rigid dynamics. Our results show it can be trained to accurately simulate the physical trajectories of dozens of objects over thousands of time steps, estimate abstract quantities such as energy, and generalize automatically to systems with different numbers and configurations of objects and relations. Our interaction network implementation is the first general-purpose, learnable physics engine, and a powerful general framework for reasoning about object and relations in a wide variety of complex real-world domains.

Published in NIPS 2016

References in corpus (2)

• Deep Learning in Neural Networks: An Overview
• To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction

Cited by in corpus (229)

• Geometric deep learning: going beyond Euclidean data
• Graph Neural Networks: A Review of Methods and Applications
• MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding
• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows
• Discovering physical concepts with neural networks
• Hamiltonian Neural Networks
• Learning to Simulate Complex Physics with Graph Networks
• Learning Deep Generative Models of Graphs
• How Powerful are Graph Neural Networks?
• Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
• Discovering Symbolic Models from Deep Learning with Inductive Biases
• Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case
• Neural Relational Inference for Interacting Systems
• Benchmarking Graph Neural Networks
• Learning to Predict the Cosmological Structure Formation
• Object-Centric Learning with Slot Attention
• Multi-view Graph Contrastive Representation Learning for Drug-Drug Interaction Prediction
• Deep Learning on Graphs: A Survey
• Graph Neural Networks: Taxonomy, Advances and Trends
• Non-local Neural Networks
• Universal Invariant and Equivariant Graph Neural Networks
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Supervised Community Detection with Line Graph Neural Networks
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• What Can Neural Networks Reason About?
• SCAN: Self-and-Collaborative Attention Network for Video Person Re-identification
• Temporal Graph Networks for Deep Learning on Dynamic Graphs
• Graph Kernels: A Survey
• Covariant Compositional Networks For Learning Graphs
• Hamiltonian Graph Networks with ODE Integrators
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
• CLEVRER: CoLlision Events for Video REpresentation and Reasoning
• Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data
• Symplectic Recurrent Neural Networks
• Relation Networks for Object Detection
• Lagrangian Neural Networks
• Graph Neural Networks Accelerated Molecular Dynamics
• An End-to-End Differentiable Framework for Contact-Aware Robot Design
• Explainability Techniques for Graph Convolutional Networks
• A Survey of Deep Learning for Scientific Discovery
• A Perspective on Deep Learning for Molecular Modeling and Simulations
• Modeling human intuitions about liquid flow with particle-based simulation
• DIG: A Turnkey Library for Diving into Graph Deep Learning Research
• MERLOT: Multimodal Neural Script Knowledge Models
• Generalization and Representational Limits of Graph Neural Networks
• Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model
• Charged particle tracking via edge-classifying interaction networks
• Symplectic Neural Networks in Taylor Series Form for Hamiltonian Systems
• Supervised Jet Clustering with Graph Neural Networks for Lorentz Boosted Bosons
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
• On the Binding Problem in Artificial Neural Networks
• Learning Compositional Koopman Operators for Model-Based Control
• Causal Discovery in Physical Systems from Videos
• Graph Convolutional Neural Networks with Node Transition Probability-based Message Passing and DropNode Regularization
• How Attentive are Graph Attention Networks?
• Comparing machine learning techniques for predicting glassy dynamics
• Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
• Learning A Physical Long-term Predictor
• Pragmatic inference and visual abstraction enable contextual flexibility during visual communication
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
• Differentiable Molecular Simulations for Control and Learning
• Graph Convolutional Reinforcement Learning
• Entity Abstraction in Visual Model-Based Reinforcement Learning
• Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAs
• Learning Symbolic Physics with Graph Networks
• Interactive Differentiable Simulation
• IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• Spatial-Temporal Relation Networks for Multi-Object Tracking
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
• Reasoning Visual Dialogs with Structural and Partial Observations
• Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations
• An Explicitly Relational Neural Network Architecture
• Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact
• What graph neural networks cannot learn: depth vs width
• Hybrid Quantum-Classical Graph Convolutional Network
• Visual Grounding of Learned Physical Models
• Building powerful and equivariant graph neural networks with structural message-passing
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Graph Element Networks: adaptive, structured computation and memory
• The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning
• ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind
• Learning Object Relation Graph and Tentative Policy for Visual Navigation
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• Pointer Graph Networks
• How hard is to distinguish graphs with graph neural networks?
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
• Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels
• Evaluating Logical Generalization in Graph Neural Networks
• AliGraph: A Comprehensive Graph Neural Network Platform
• A Perspective on Objects and Systematic Generalization in Model-Based RL
• Reliable Graph Neural Networks for Drug Discovery Under Distributional Shift
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• Discovering Nonlinear Relations with Minimum Predictive Information Regularization
• RigNet: Neural Rigging for Articulated Characters
• Active Learning of Abstract Plan Feasibility
• Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gap
• Physion: Evaluating Physical Prediction from Vision in Humans and Machines
• Text Level Graph Neural Network for Text Classification
• Factor Graph Neural Network
• Estimating Mass Distribution of Articulated Objects using Non-prehensile Manipulation
• Graph Generative Models for Fast Detector Simulations in High Energy Physics
• Learning Physical Constraints with Neural Projections
• Propagation Networks for Model-Based Control Under Partial Observation
• STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
• On the Equivalence Between Temporal and Static Graph Representations for Observational Predictions
• Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects
• Gated Graph Recursive Neural Networks for Molecular Property Prediction
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making
• Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving
• Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization
• Learning to Prove from Synthetic Theorems
• Relational Graph Learning for Crowd Navigation
• Learning continuous-time PDEs from sparse data with graph neural networks
• Efficient Coarse-to-Fine Non-Local Module for the Detection of Small Objects
• Molecule Property Prediction and Classification with Graph Hypernetworks
• Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• Occlusion resistant learning of intuitive physics from videos
• Reinforcement Learning for Adaptive Mesh Refinement
• Using Learnable Physics for Real-Time Exercise Form Recommendations
• Forward Prediction for Physical Reasoning
• Understanding complex crowd dynamics with generative neural simulators
• Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models
• Modeling Attention Flow on Graphs
• SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation
• When2com: Multi-Agent Perception via Communication Graph Grouping
• Counting Substructures with Higher-Order Graph Neural Networks: Possibility and Impossibility Results
• Analogical Reasoning for Visually Grounded Language Acquisition
• Graph Neural Network Architecture Search for Molecular Property Prediction
• Demand Forecasting from Spatiotemporal Data with Graph Networks and Temporal-Guided Embedding
• Compositional Video Synthesis with Action Graphs
• Active World Model Learning with Progress Curiosity
• Explainable Deep Relational Networks for Predicting Compound-Protein Affinities and Contacts
• Visual Physics: Discovering Physical Laws from Videos
• Generalization and Robustness Implications in Object-Centric Learning
• A First Principles Approach for Data-Efficient System Identification of Spring-Rod Systems via Differentiable Physics Engines
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Neural Vortex Method: from Finite Lagrangian Particles to Infinite Dimensional Eulerian Dynamics
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks
• A Neural-Symbolic Architecture for Inverse Graphics Improved by Lifelong Meta-Learning
• Generalization bounds for graph convolutional neural networks via Rademacher complexity
• Equivariant Entity-Relationship Networks
• Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control
• Lane Attention: Predicting Vehicles' Moving Trajectories by Learning Their Attention over Lanes
• Graph Neural Processes: Towards Bayesian Graph Neural Networks
• Intrinsic Motivation Driven Intuitive Physics Learning using Deep Reinforcement Learning with Intrinsic Reward Normalization
• Putting visual object recognition in context
• Integrating Physics and Topology in Neural Networks for Learning Rigid Body Dynamics
• Learning Generalizable Physical Dynamics of 3D Rigid Objects
• Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
• Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks
• Unsupervised Intuitive Physics from Past Experiences
• Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation
• Visual Semantic Information Pursuit: A Survey
• RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Search For Deep Graph Neural Networks
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• Deep learning of material transport in complex neurite networks
• Particle Graph Autoencoders and Differentiable, Learned Energy Mover's Distance
• Faster Policy Learning with Continuous-Time Gradients
• Locality Preserving Dense Graph Convolutional Networks with Graph Context-Aware Node Representations
• Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
• Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling
• Unsupervised Resource Allocation with Graph Neural Networks
• Time-Reversal Symmetric ODE Network
• Universal Approximation of Functions on Sets
• SHADOWCAST: Controllable Graph Generation
• Who2com: Collaborative Perception via Learnable Handshake Communication
• HR-RCNN: Hierarchical Relational Reasoning for Object Detection
• Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution
• Time-Series Event Prediction with Evolutionary State Graph
• Improved Structural Discovery and Representation Learning of Multi-Agent Data
• Relational State-Space Model for Stochastic Multi-Object Systems
• Causal Learning by a Robot with Semantic-Episodic Memory in an Aesop's Fable Experiment
• Typed Graph Networks
• Structured agents for physical construction
• Computing Steiner Trees using Graph Neural Networks
• RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting
• Dynamic Relational Inference in Multi-Agent Trajectories
• Identifying Physical Law of Hamiltonian Systems via Meta-Learning
• Relational Reasoning Network (RRN) for Anatomical Landmarking
• Generative Deep Learning Model for a Multi-level Nano-Optic Broadband Power Splitter
• Object-Oriented Dynamics Learning through Multi-Level Abstraction
• Deep Learning on Attributed Graphs: A Journey from Graphs to Their Embeddings and Back
• Lift-the-flap: what, where and when for context reasoning
• Amanuensis: The Programmer's Apprentice
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals
• End-to-End Learning from Complex Multigraphs with Latent-Graph Convolutional Networks
• Particle Track Reconstruction using Geometric Deep Learning
• Bayes EMbedding (BEM): Refining Representation by Integrating Knowledge Graphs and Behavior-specific Networks
• Learning Topological Motion Primitives for Knot Planning
• SimuLearn: Fast and Accurate Simulator to Support Morphing Materials Design and Workflows
• Learning Transition Models with Time-delayed Causal Relations
• A FAIR and AI-ready Higgs boson decay dataset
• Improving Simulations with Symmetry Control Neural Networks
• Compositional Video Prediction
• Volumetric Transformer Networks
• Plug and Play, Model-Based Reinforcement Learning
• MagNet: Discovering Multi-agent Interaction Dynamics using Neural Network
• The Role of Isomorphism Classes in Multi-Relational Datasets
• Can NetGAN be improved on short random walks?
• Irregular Convolutional Auto-Encoder on Point Clouds
• Weighted Graph Nodes Clustering via Gumbel Softmax
• Biological Blueprints for Next Generation AI Systems
• Spring-Rod System Identification via Differentiable Physics Engine
• Learning on Random Balls is Sufficient for Estimating (Some) Graph Parameters
• Deep Convolution for Irregularly Sampled Temporal Point Clouds
• Active Learning of Neural Collision Handler for Complex 3D Mesh Deformations
• Human-Guided Learning of Column Networks: Augmenting Deep Learning with Advice
• Depthwise Non-local Module for Fast Salient Object Detection Using a Single Thread
• Graph-based Joint Pandemic Concern and Relation Extraction on Twitter
• Conditional Object-Centric Learning from Video
• Nonseparable Symplectic Neural Networks
• Discovering Latent Representations of Relations for Interacting Systems