Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

Abstract

What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.

References in corpus (6)

• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Universal Correspondence Network
• Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing
• Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching
• End-to-End Learning of Semantic Grasping
• Semantic Segmentation from Limited Training Data

Cited by in corpus (35)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation
• Transporter Networks: Rearranging the Visual World for Robotic Manipulation
• Challenges for Monocular 6D Object Pose Estimation in Robotics
• DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Grasping
• Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints
• Learning Densities in Feature Space for Reliable Segmentation of Indoor Scenes
• Graph-Structured Visual Imitation
• S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency
• Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics
• Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly
• Self-supervised 6D Object Pose Estimation for Robot Manipulation
• Online Object Representations with Contrastive Learning
• Learning to Navigate Using Mid-Level Visual Priors
• The Treachery of Images: Bayesian Scene Keypoints for Deep Policy Learning in Robotic Manipulation
• Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks
• Model-Based Inverse Reinforcement Learning from Visual Demonstrations
• FilterReg: Robust and Efficient Probabilistic Point-Set Registration using Gaussian Filter and Twist Parameterization
• Semantic Correspondence via 2D-3D-2D Cycle
• Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
• DIJE: Dense Image Jacobian Estimation for Robust Robotic Self-Recognition and Visual Servoing
• Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
• A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes
• Active Perception and Representation for Robotic Manipulation
• Object-Agnostic Suction Grasp Affordance Detection in Dense Cluster Using Self-Supervised Learning.docx
• Machine Learning for Robotic Manipulation
• Visual Task Progress Estimation with Appearance Invariant Embeddings for Robot Control and Planning
• MMGSD: Multi-Modal Gaussian Shape Descriptors for Correspondence Matching in 1D and 2D Deformable Objects
• Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine
• Combining Deep Learning and Verification for Precise Object Instance Detection
• Human Correspondence Consensus for 3D Object Semantic Understanding
• Recognizing Object Affordances to Support Scene Reasoning for Manipulation Tasks