Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects

Abstract

The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of such objects. We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects with sufficient accuracy to find and grasp them securely. We leverage NeRF's view-independent learned density, place lights to increase specular reflections, and perform a transparency-aware depth-rendering that we feed into the Dex-Net grasp planner. We show how additional lights create specular reflections that improve the quality of the depth map, and test a setup for a robot workcell equipped with an array of cameras to perform transparent object manipulation. We also create synthetic and real datasets of transparent objects in real-world settings, including singulated objects, cluttered tables, and the top rack of a dishwasher. In each setting we show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90% and 100% grasp success rates in physical experiments on an ABB YuMi, on objects where baseline methods fail.

arXiv comment11 pages, 9 figures, to be published in the Conference on Robot Learning (CoRL) 2021

References in corpus (8)

• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
• Depth-supervised NeRF: Fewer Views and Faster Training for Free
• Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter
• GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter
• RGB-D Local Implicit Function for Depth Completion of Transparent Objects

Cited by in corpus (7)

• TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline
• Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds
• Challenges for Monocular 6D Object Pose Estimation in Robotics
• FDCT: Fast Depth Completion for Transparent Objects
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
• ReFlow6D: Refraction-Guided Transparent Object 6D Pose Estimation via Intermediate Representation Learning