The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?

Abstract

A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance.

10 pages, published at the 1st Annual Conference on Robot Learning (CoRL), Code and dataset available at: https://lasr.org/research/feeling-of-success

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty
• Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
• Connecting Look and Feel: Associating the visual and tactile properties of physical materials

Cited by in corpus (21)

• More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors
• From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor
• FingerVision Tactile Sensor Design and Slip Detection Using Convolutional LSTM Network
• Visual-Tactile Cross-Modal Data Generation using Residue-Fusion GAN with Feature-Matching and Perceptual Losses
• Manipulation by Feel: Touch-Based Control with Deep Predictive Models
• Towards Learning to Detect and Predict Contact Events on Vision-based Tactile Sensors
• In-Hand Object Stabilization by Independent Finger Control
• Learning Awareness Models
• Material Segmentation of Multi-View Satellite Imagery
• 3D Shape Perception from Monocular Vision, Touch, and Shape Priors
• ReSkin: versatile, replaceable, lasting tactile skins
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Bayesian Grasp: Robotic visual stable grasp based on prior tactile knowledge
• Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception
• Optimal Deep Learning for Robot Touch
• Sensor Fusion: Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data
• Learn and Transfer Knowledge of Preferred Assistance Strategies in Semi-autonomous Telemanipulation
• Comparing Piezoresistive Substrates for Tactile Sensing in Dexterous Hands
• Learning to See before Learning to Act: Visual Pre-training for Manipulation