Interaction Networks for Learning about Objects, Relations and Physics

Abstract

Reasoning about objects, relations, and physics is central to human intelligence, and a key goal of artificial intelligence. Here we introduce the interaction network, a model which can reason about how objects in complex systems interact, supporting dynamical predictions, as well as inferences about the abstract properties of the system. Our model takes graphs as input, performs object- and relation-centric reasoning in a way that is analogous to a simulation, and is implemented using deep neural networks. We evaluate its ability to reason about several challenging physical domains: n-body problems, rigid-body collision, and non-rigid dynamics. Our results show it can be trained to accurately simulate the physical trajectories of dozens of objects over thousands of time steps, estimate abstract quantities such as energy, and generalize automatically to systems with different numbers and configurations of objects and relations. Our interaction network implementation is the first general-purpose, learnable physics engine, and a powerful general framework for reasoning about object and relations in a wide variety of complex real-world domains.

Published in NIPS 2016

References in corpus (1)

• Deep Learning in Neural Networks: An Overview

Cited by in corpus (94)

• Covariant Compositional Networks For Learning Graphs
• Hamiltonian Graph Networks with ODE Integrators
• Graph Neural Networks Accelerated Molecular Dynamics
• An End-to-End Differentiable Framework for Contact-Aware Robot Design
• Explainability Techniques for Graph Convolutional Networks
• MERLOT: Multimodal Neural Script Knowledge Models
• Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model
• Generalization and Representational Limits of Graph Neural Networks
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
• On the Binding Problem in Artificial Neural Networks
• Learning A Physical Long-term Predictor
• Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
• Accelerated Charged Particle Tracking with Graph Neural Networks on FPGAs
• Learning Symbolic Physics with Graph Networks
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• Spatial-Temporal Relation Networks for Multi-Object Tracking
• Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations
• ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
• Reasoning Visual Dialogs with Structural and Partial Observations
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Hybrid Quantum-Classical Graph Convolutional Network
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Learning Object Relation Graph and Tentative Policy for Visual Navigation
• Structured Object-Aware Physics Prediction for Video Modeling and Planning
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
• AliGraph: A Comprehensive Graph Neural Network Platform
• A Perspective on Objects and Systematic Generalization in Model-Based RL
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• Discovering Nonlinear Relations with Minimum Predictive Information Regularization
• RigNet: Neural Rigging for Articulated Characters
• Active Learning of Abstract Plan Feasibility
• Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gap
• Factor Graph Neural Network
• Text Level Graph Neural Network for Text Classification
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
• Learning to Prove from Synthetic Theorems
• Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
• Molecule Property Prediction and Classification with Graph Hypernetworks
• Using Learnable Physics for Real-Time Exercise Form Recommendations
• Occlusion resistant learning of intuitive physics from videos
• Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs
• When2com: Multi-Agent Perception via Communication Graph Grouping
• A First Principles Approach for Data-Efficient System Identification of Spring-Rod Systems via Differentiable Physics Engines
• Analogical Reasoning for Visually Grounded Language Acquisition
• Graph Neural Network Architecture Search for Molecular Property Prediction
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Active World Model Learning with Progress Curiosity
• Explainable Deep Relational Networks for Predicting Compound-Protein Affinities and Contacts
• Visual Physics: Discovering Physical Laws from Videos
• Generalization bounds for graph convolutional neural networks via Rademacher complexity
• Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning
• Intrinsic Motivation Driven Intuitive Physics Learning using Deep Reinforcement Learning with Intrinsic Reward Normalization
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning
• Learning Generalizable Physical Dynamics of 3D Rigid Objects
• Visual Semantic Information Pursuit: A Survey
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Unsupervised Intuitive Physics from Past Experiences
• Unsupervised Object Keypoint Learning using Local Spatial Predictability
• Search For Deep Graph Neural Networks
• RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks
• Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation
• Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks
• Universal Approximation of Functions on Sets
• Improved Structural Discovery and Representation Learning of Multi-Agent Data
• Relational State-Space Model for Stochastic Multi-Object Systems
• Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling
• Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution
• Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
• Typed Graph Networks
• Deep learning of material transport in complex neurite networks
• Unsupervised Resource Allocation with Graph Neural Networks
• Causal Learning by a Robot with Semantic-Episodic Memory in an Aesop's Fable Experiment
• Structured agents for physical construction
• Identifying Physical Law of Hamiltonian Systems via Meta-Learning
• RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting
• Deep Learning on Attributed Graphs: A Journey from Graphs to Their Embeddings and Back
• SimuLearn: Fast and Accurate Simulator to Support Morphing Materials Design and Workflows
• Learning Transition Models with Time-delayed Causal Relations
• Learning Topological Motion Primitives for Knot Planning
• Particle Track Reconstruction using Geometric Deep Learning
• Computing Steiner Trees using Graph Neural Networks
• Plug and Play, Model-Based Reinforcement Learning
• Human-Guided Learning of Column Networks: Augmenting Deep Learning with Advice
• Depthwise Non-local Module for Fast Salient Object Detection Using a Single Thread
• Spring-Rod System Identification via Differentiable Physics Engine
• Learning on Random Balls is Sufficient for Estimating (Some) Graph Parameters
• Active Learning of Neural Collision Handler for Complex 3D Mesh Deformations
• Weighted Graph Nodes Clustering via Gumbel Softmax
• Volumetric Transformer Networks
• The Role of Isomorphism Classes in Multi-Relational Datasets
• Irregular Convolutional Auto-Encoder on Point Clouds
• Biological Blueprints for Next Generation AI Systems
• Graph-based Joint Pandemic Concern and Relation Extraction on Twitter