gradSim: Differentiable simulation for system identification and visuomotor control

Abstract

We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph -- spanning from the dynamics and through the rendering process -- enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.

ICLR 2021. Project page (and a dynamic web version of the article): https://gradsim.github.io

References in corpus (13)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Open3D: A Modern Library for 3D Data Processing
• A Review of automatic differentiation and its efficient implementation
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Accelerating 3D Deep Learning with PyTorch3D
• Hamiltonian Graph Networks with ODE Integrators
• Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
• Learning A Physical Long-term Predictor
• Scalable Differentiable Physics for Learning and Control
• Interactive Differentiable Simulation
• Tangent: Automatic Differentiation Using Source Code Transformation in Python
• Neural Dynamical Systems: Balancing Structure and Flexibility in Physical Prediction
• Identifying Mechanical Models through Differentiable Simulations

Cited by in corpus (11)

• DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact
• Differentiable solver for time-dependent deformation problems with contact
• How Will It Drape Like? Capturing Fabric Mechanics from Depth Images
• DiffSDFSim: Differentiable Rigid-Body Dynamics With Implicit Shapes
• Neurally Integrated Finite Elements for Differentiable Elasticity on Evolving Domains
• MGPBD: A Multigrid Accelerated Global XPBD Solver
• Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models
• Follow the Gradient: Crossing the Reality Gap using Differentiable Physics (RealityGrad)
• Vision-based system identification and 3D keypoint discovery using dynamics constraints
• Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos
• DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks