Data-Efficient Learning for Sim-to-Real Robotic Grasping using Deep Point Cloud Prediction Networks

Abstract

Training a deep network policy for robot manipulation is notoriously costly and time consuming as it depends on collecting a significant amount of real world data. To work well in the real world, the policy needs to see many instances of the task, including various object arrangements in the scene as well as variations in object geometry, texture, material, and environmental illumination. In this paper, we propose a method that learns to perform table-top instance grasping of a wide variety of objects while using no real world grasping data, outperforming the baseline using 2.5D shape by 10%. Our method learns 3D point cloud of object, and use that to train a domain-invariant grasping policy. We formulate the learning process as a two-step procedure: 1) Learning a domain-invariant 3D shape representation of objects from about 76K episodes in simulation and about 530 episodes in the real world, where each episode lasts less than a minute and 2) Learning a critic grasping policy in simulation only based on the 3D shape representation from step 1. Our real world data collection in step 1 is both cheaper and faster compared to existing approaches as it only requires taking multiple snapshots of the scene using a RGBD camera. Finally, the learned 3D representation is not specific to grasping, and can potentially be used in other interaction tasks.

References in corpus (12)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Learning Representations and Generative Models for 3D Point Clouds
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
• Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction
• Shape Completion Enabled Robotic Grasping
• 3D Shape Perception from Monocular Vision, Touch, and Shape Priors
• Parsing Geometry Using Structure-Aware Shape Templates

Cited by in corpus (6)

• Hybrid Robotic Grasping with a Soft Multimodal Gripper and a Deep Multistage Learning Scheme
• Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
• Learning Fast Adaptation with Meta Strategy Optimization
• RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer
• Towards Precise Model-free Robotic Grasping with Sim-to-Real Transfer Learning
• GPR: Grasp Pose Refinement Network for Cluttered Scenes