Vision-based Robotic Grasping From Object Localization, Object Pose Estimation to Grasp Estimation for Parallel Grippers: A Review

Abstract

This paper presents a comprehensive survey on vision-based robotic grasping. We conclude three key tasks during vision-based robotic grasping, which are object localization, object pose estimation and grasp estimation. In detail, the object localization task contains object localization without classification, object detection and object instance segmentation. This task provides the regions of the target object in the input data. The object pose estimation task mainly refers to estimating the 6D object pose and includes correspondence-based methods, template-based methods and voting-based methods, which affords the generation of grasp poses for known objects. The grasp estimation task includes 2D planar grasp methods and 6DoF grasp methods, where the former is constrained to grasp from one direction. These three tasks could accomplish the robotic grasping with different combinations. Lots of object pose estimation methods need not object localization, and they conduct object localization and object pose estimation jointly. Lots of grasp estimation methods need not object localization and object pose estimation, and they conduct grasp estimation in an end-to-end manner. Both traditional methods and latest deep learning-based methods based on the RGB-D image inputs are reviewed elaborately in this survey. Related datasets and comparisons between state-of-the-art methods are summarized as well. In addition, challenges about vision-based robotic grasping and future directions in addressing these challenges are also pointed out.

This is a pre-print of an article published in Artificial Intelligence Review. The final authenticated version is available online at: https://doi.org/10.1007/s10462-020-09888-5. Related refs are summarized at: https://github.com/GeorgeDu/vision-based-robotic-grasping

References in corpus (21)

• SSD: Single Shot MultiBox Detector
• ORB-SLAM: a Versatile and Accurate Monocular SLAM System
• Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration
• Data-Driven Grasp Synthesis - A Survey
• Salient Object Detection: A Survey
• A Survey on Instance Segmentation: State of the art
• Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation
• 3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration
• RGBD Salient Object Detection via Deep Fusion
• PointNetGPD: Detecting Grasp Configurations from Point Sets
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• DeepICP: An End-to-End Deep Neural Network for 3D Point Cloud Registration
• A Review of Object Detection Models based on Convolutional Neural Network
• SegICP: Integrated Deep Semantic Segmentation and Pose Estimation
• Robust Grasp Planning Over Uncertain Shape Completions
• A Transfer Learning Approach to Cross-Modal Object Recognition: From Visual Observation to Robotic Haptic Exploration
• DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Grasping
• When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
• Online Self-Supervised Learning for Object Picking: Detecting Optimum Grasping Position using a Metric Learning Approach
• Accurate 6D Object Pose Estimation by Pose Conditioned Mesh Reconstruction
• A Summary of the 4th International Workshop on Recovering 6D Object Pose

Cited by in corpus (11)

• Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchards
• Visuo-Haptic Object Perception for Robots: An Overview
• Enabling Building Information Model-Driven Human-Robot Collaborative Construction Workflows with Closed-Loop Digital Twins
• Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective
• Category-Association Based Similarity Matching for Novel Object Pick-and-Place Task
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• CylinderTag: An Accurate and Flexible Marker for Cylinder-Shape Objects Pose Estimation Based on Projective Invariants
• Incipient Slip-Based Rotation Measurement via Visuotactile Sensing During In-Hand Object Pivoting
• Role of Uncertainty in Model Development and Control Design for a Manufacturing Process
• Simple Kinesthetic Haptics for Object Recognition
• LocaliseBot: Multi-view 3D object localisation with differentiable rendering for robot grasping