Vid2Param: Modelling of Dynamics Parameters from Video

Abstract

Videos provide a rich source of information, but it is generally hard to extract dynamical parameters of interest. Inferring those parameters from a video stream would be beneficial for physical reasoning. Robots performing tasks in dynamic environments would benefit greatly from understanding the underlying environment motion, in order to make future predictions and to synthesize effective control policies that use this inductive bias. Online physical reasoning is therefore a fundamental requirement for robust autonomous agents. When the dynamics involves multiple modes (due to contacts or interactions between objects) and sensing must proceed directly from a rich sensory stream such as video, then traditional methods for system identification may not be well suited. We propose an approach wherein fast parameter estimation can be achieved directly from video. We integrate a physically based dynamics model with a recurrent variational autoencoder, by introducing an additional loss to enforce desired constraints. The model, which we call Vid2Param, can be trained entirely in simulation, in an end-to-end manner with domain randomization, to perform online system identification, and make probabilistic forward predictions of parameters of interest. This enables the resulting model to encode parameters such as position, velocity, restitution, air drag and other physical properties of the system. We illustrate the utility of this in physical experiments wherein a PR2 robot with a velocity constrained arm must intercept an unknown bouncing ball with partly occluded vision, by estimating the physical parameters of this ball directly from the video trace after the ball is released.

Accepted as a journal paper at IEEE Robotics and Automation Letters (RA-L)

References in corpus (13)

• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• A Recurrent Latent Variable Model for Sequential Data
• Neural Lander: Stable Drone Landing Control using Learned Dynamics
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Stochastic Adversarial Video Prediction
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Flexible Neural Representation for Physics Prediction
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• The ORCA Hub: Explainable Offshore Robotics through Intelligent Interfaces
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Interpretable Latent Spaces for Learning from Demonstration
• Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

Cited by in corpus (3)

• A First Principles Approach for Data-Efficient System Identification of Spring-Rod Systems via Differentiable Physics Engines
• Vision-based system identification and 3D keypoint discovery using dynamics constraints
• Spring-Rod System Identification via Differentiable Physics Engine