Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video

Abstract

We propose a model that is able to perform unsupervised physical parameter estimation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states or objects are not available. Existing physical scene understanding methods require either object state supervision, or do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through a physics-as-inverse-graphics approach that brings together vision-as-inverse-graphics and differentiable physics engines, enabling objects and explicit state and velocity representations to be discovered. This framework allows us to perform long term extrapolative video prediction, as well as vision-based model-predictive control. Our approach significantly outperforms related unsupervised methods in long-term future frame prediction of systems with interacting objects (such as ball-spring or 3-body gravitational systems), due to its ability to build dynamics into the model as an inductive bias. We further show the value of this tight vision-physics integration by demonstrating data-efficient learning of vision-actuated model-based control for a pendulum system. We also show that the controller's interpretability provides unique capabilities in goal-driven control and physical reasoning for zero-data adaptation.

arXiv comment11 pages

References in corpus (21)

• Continuous control with deep reinforcement learning
• Unsupervised Learning of Video Representations using LSTMs
• Deep Convolutional Inverse Graphics Network
• Learning Latent Dynamics for Planning from Pixels
• Distributed Distributional Deterministic Policy Gradients
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Multi-Object Representation Learning with Iterative Variational Inference
• MONet: Unsupervised Scene Decomposition and Representation
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Simple random search provides a competitive approach to reinforcement learning
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Learning to Decompose and Disentangle Representations for Video Prediction
• One-Shot Generalization in Deep Generative Models
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks
• Reasoning About Physical Interactions with Object-Oriented Prediction and Planning
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Efficient inference in occlusion-aware generative models of images
• SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control
• Unsupervised Discovery of Parts, Structure, and Dynamics
• Object-Oriented Dynamics Predictor

Cited by in corpus (6)

• PDE-Driven Spatiotemporal Disentanglement
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling
• Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Control
• Vision-based system identification and 3D keypoint discovery using dynamics constraints
• Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videos
• Discovering Latent Representations of Relations for Interacting Systems