Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

Abstract

When operating in unstructured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables. Mechanical Search describes the class of tasks where the goal is to locate and extract a known target object. In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin. The robot uses an RGBD perception system and control policies to iteratively select, parameterize, and perform one of 3 actions -- push, suction, grasp -- until the target object is extracted, or either a time limit is exceeded, or no high confidence push or grasp is available. We present a study of 5 algorithmic policies for mechanical search, with 15,000 simulated trials and 300 physical trials for heaps ranging from 10 to 20 objects. Results suggest that success can be achieved in this long-horizon task with algorithmic policies in over 95% of instances and that the number of actions required scales approximately linearly with the size of the heap. Code and supplementary material can be found at http://ai.stanford.edu/mech-search .

arXiv commentTo appear in IEEE International Conference on Robotics and Automation (ICRA), 2019. 9 pages with 4 figures

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Matching Networks for One Shot Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Interactive Perception: Leveraging Action in Perception and Perception in Action
• One-Shot Imitation Learning
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• Learning to Refine Object Segments
• Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
• End-to-End Learning of Semantic Grasping
• Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision
• Do What I Want, Not What I Did: Imitation of Skills by Planning Sequences of Actions

Cited by in corpus (18)

• Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangement
• Learning Deformable Object Manipulation from Expert Demonstrations
• MQA: Answering the Question via Robotic Manipulation
• X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions
• Learning to Dexterously Pick or Separate Tangled-Prone Objects for Industrial Bin Picking
• Object Finding in Cluttered Scenes Using Interactive Perception
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter
• Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• A Joint Network for Grasp Detection Conditioned on Natural Language Commands
• A Deep Learning Approach to Grasping the Invisible
• Collision-Aware Target-Driven Object Grasping in Constrained Environments
• Mechanical Search on Shelves using Lateral Access X-RAY
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Learning Visual Affordances with Target-Orientated Deep Q-Network to Grasp Objects by Harnessing Environmental Fixtures
• Implicit Shape Model Trees: Recognition of 3-D Indoor Scenes and Prediction of Object Poses for Mobile Robots
• Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms
• Attribute-Based Robotic Grasping with One-Grasp Adaptation