Visual Grounding of Learned Physical Models

Abstract

Humans intuitively recognize objects' physical properties and predict their motion, even when the objects are engaged in complicated interactions. The abilities to perform physical reasoning and to adapt to new environments, while intrinsic to humans, remain challenging to state-of-the-art computational models. In this work, we present a neural model that simultaneously reasons about physics and makes future predictions based on visual and dynamics priors. The visual prior predicts a particle-based representation of the system from visual observations. An inference module operates on those particles, predicting and refining estimates of particle locations, object states, and physical parameters, subject to the constraints imposed by the dynamics prior, which we refer to as visual grounding. We demonstrate the effectiveness of our method in environments involving rigid objects, deformable materials, and fluids. Experiments show that our model can infer the physical properties within a few observations, which allows the model to quickly adapt to unseen scenarios and make accurate predictions into the future.

The second and the third authors contributed equally to this paper, and are listed in alphabetical order. Project Page: http://visual-physics-grounding.csail.mit.edu/

References in corpus (14)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Interaction Networks for Learning about Objects, Relations and Physics
• Recurrent World Models Facilitate Policy Evolution
• Learning to Simulate Complex Physics with Graph Networks
• Learning Latent Dynamics for Planning from Pixels
• Graph networks as learnable physics engines for inference and control
• Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Flexible Neural Representation for Physics Prediction
• Visual Interaction Networks
• Modeling human intuitions about liquid flow with particle-based simulation
• Learning Compositional Koopman Operators for Model-Based Control
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning

Cited by in corpus (5)

• Graph neural networks for simulating crack coalescence and propagation in brittle materials
• MPMNet: A Data-Driven MPM Framework for Dynamic Fluid-Solid Interaction
• Discovering Latent Representations of Relations for Interacting Systems
• Pushing it out of the Way: Interactive Visual Navigation
• Continuous Perception for Classifying Shapes and Weights of Garmentsfor Robotic Vision Applications