RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter

Abstract

Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic data sets possible. We evaluate our approach on two challenging data sets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task, and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SSD: Single Shot MultiBox Detector
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Fully Convolutional Networks for Semantic Segmentation
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Mask R-CNN
• NimbRo Rescue: Solving Disaster-Response Tasks through Mobile Manipulation Robot Momaro
• A Summary of Team MIT's Approach to the Amazon Picking Challenge 2015
• The ACRV Picking Benchmark (APB): A Robotic Shelf Picking Benchmark to Foster Reproducible Research
• Iterative Visual Recognition for Learning Based Randomized Bin-Picking

Cited by in corpus (26)

• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing
• SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping
• From SLAM to Situational Awareness: Challenges and Survey
• Robust Grasp Planning Over Uncertain Shape Completions
• FPCC: Fast Point Cloud Clustering based Instance Segmentation for Industrial Bin-picking
• Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands
• Universal image segmentation for optical identification of 2D materials
• PIG-Net: Inception based Deep Learning Architecture for 3D Point Cloud Segmentation
• Reinforced active learning for image segmentation
• Boosting Deep Open World Recognition by Clustering
• FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything
• Value Iteration Networks on Multiple Levels of Abstraction
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• A Graph-based Optimization Framework for Hand-Eye Calibration for Multi-Camera Setups
• Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data
• Weakly-Supervised Object Detection Learning through Human-Robot Interaction
• Semantic Segmentation from Limited Training Data
• Fast Region Proposal Learning for Object Detection for Robotics
• Straight to Shapes++: Real-time Instance Segmentation Made More Accurate
• A Review on Object Pose Recovery: from 3D Bounding Box Detectors to Full 6D Pose Estimators
• Importance of Self-Consistency in Active Learning for Semantic Segmentation
• A vision based system for underwater docking
• LiDAR-assisted Large-scale Privacy Protection in Street-view Cycloramas
• Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly
• Team NimbRo at MBZIRC 2017: Autonomous Valve Stem Turning using a Wrench