End-to-End Learning of Semantic Grasping

Abstract

We consider the task of semantic robotic grasping, in which a robot picks up an object of a user-specified class using only monocular images. Inspired by the two-stream hypothesis of visual reasoning, we present a semantic grasping framework that learns object detection, classification, and grasp planning in an end-to-end fashion. A "ventral stream" recognizes object class while a "dorsal stream" simultaneously interprets the geometric relationships necessary to execute successful grasps. We leverage the autonomous data collection capabilities of robots to obtain a large self-supervised dataset for training the dorsal stream, and use semi-supervised label propagation to train the ventral stream with only a modest amount of human supervision. We experimentally show that our approach improves upon grasping systems whose components are not learned end-to-end, including a baseline method that uses bounding box detection. Furthermore, we show that jointly training our model with auxiliary data consisting of non-semantic grasping data, as well as semantically labeled images without grasp actions, has the potential to substantially improve semantic grasping performance.

arXiv comment14 pages

References in corpus (1)

• Distilling the Knowledge in a Neural Network

Cited by in corpus (14)

• Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
• Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Grasping
• X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions
• Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision
• Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System
• Learning a Generative Transition Model for Uncertainty-Aware Robotic Manipulation
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• A Deep Learning Approach to Grasping the Invisible
• REGRAD: A Large-Scale Relational Grasp Dataset for Safe and Object-Specific Robotic Grasping in Clutter