Visual Interaction Networks
arXiv:1706.01433
Abstract
From just a glance, humans can make rich predictions about the future state of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains and require direct measurements of the underlying states. We introduce the Visual Interaction Network, a general-purpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions and dynamics, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. Our results demonstrate that the perceptual module and the object-based dynamics predictor module can induce factored latent representations that support accurate dynamical predictions. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.
References in corpus (5)
Cited by in corpus (37)
- Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows
- Deep Learning: A Critical Appraisal
- FiLM: Visual Reasoning with a General Conditioning Layer
- Deformable ConvNets v2: More Deformable, Better Results
- Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
- Flexible Neural Representation for Physics Prediction
- Relational Deep Reinforcement Learning
- Non-local Neural Networks
- Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
- SCAN: Self-and-Collaborative Attention Network for Video Person Re-identification
- Relation Networks for Object Detection
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning
- Differentiable Physics-informed Graph Networks
- SPNets: Differentiable Fluid Dynamics for Deep Neural Networks
- Graph Convolutional Reinforcement Learning
- IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning
- Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
- Visual Grounding of Learned Physical Models
- DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions
- Factorised Neural Relational Inference for Multi-Interaction Systems
- Novel Video Prediction for Large-scale Scene using Optical Flow
- Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
- Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces
- Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions
- Occlusion resistant learning of intuitive physics from videos
- LinkNet: Relational Embedding for Scene Graph
- Long-Term Image Boundary Prediction
- Learning Generalizable Physical Dynamics of 3D Rigid Objects
- Learning to Play General Video-Games via an Object Embedding Network
- Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition
- Cost-Based Goal Recognition Meets Deep Learning
- Modular meta-learning in abstract graph networks for combinatorial generalization
- Object-Oriented Dynamics Learning through Multi-Level Abstraction
- Interpretable Intuitive Physics Model
- Predicting the Physical Dynamics of Unseen 3D Objects
- Compositional Video Prediction
- Learning to Transfer Visual Effects from Videos to Images