SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control

Abstract

In this work, we present an approach to deep visuomotor control using structured deep dynamics models. Our deep dynamics model, a variant of SE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an encoder-decoder structure. Unlike prior work, our dynamics model is structured: given an input scene, our network explicitly learns to segment salient parts and predict their pose-embedding along with their motion modeled as a change in the pose space due to the applied actions. We train our model using a pair of point clouds separated by an action and show that given supervision only in the form of point-wise data associations between the frames our network is able to learn a meaningful segmentation of the scene along with consistent poses. We further show that our model can be used for closed-loop control directly in the learned low-dimensional pose space, where the actions are computed by minimizing error in the pose space using gradient-based methods, similar to traditional model-based control. We present results on controlling a Baxter robot from raw depth data in simulation and in the real world and compare against two baseline deep networks. Our method runs in real-time, achieves good prediction of scene dynamics and outperforms the baseline methods on multiple control runs. Video results can be found at: https://rse-lab.cs.washington.edu/se3-structured-deep-ctrl/

8 pages, Initial submission to IEEE International Conference on Robotics and Automation (ICRA) 2018

References in corpus (3)

• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Learning Visual Servoing with Deep Features and Fitted Q-Iteration
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations

Cited by in corpus (13)

• Stochastic Adversarial Video Prediction
• SPNets: Differentiable Fluid Dynamics for Deep Neural Networks
• Dynamic Manipulation of Flexible Objects with Torque Sequence Using a Deep Neural Network
• Learning 3D Dynamic Scene Representations for Robot Manipulation
• Time Reversal as Self-Supervision
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• Unseen Object Instance Segmentation for Robotic Environments
• Task-specific Self-body Controller Acquisition by Musculoskeletal Humanoids: Application to Pedal Control in Autonomous Driving
• Learning to Manipulate Object Collections Using Grounded State Representations
• On Machine Learning and Structure for Mobile Robots
• Motion-Nets: 6D Tracking of Unknown Objects in Unseen Environments using RGB
• Sufficiently Accurate Model Learning
• Predicting the Physical Dynamics of Unseen 3D Objects