More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

Abstract

For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this paper, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors, nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6,450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at (i) estimating grasp adjustment outcomes, (ii) selecting efficient grasp adjustments for quick grasping, and (iii) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.

8 pages. Published on IEEE Robotics and Automation Letters (RAL). Website: https://sites.google.com/view/more-than-a-feeling

References in corpus (3)

• Data-Driven Grasp Synthesis - A Survey
• Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor
• The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?

Cited by in corpus (42)

• DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
• From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Ground truth force distribution for learning-based tactile sensing: a finite element approach
• Visuo-Haptic Object Perception for Robots: An Overview
• When Vision Meets Touch: A Contemporary Review for Visuotactile Sensors from the Signal Processing Perspective
• Discrete and continuous representations and processing in deep learning: Looking forward
• In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• Manipulation by Feel: Touch-Based Control with Deep Predictive Models
• Haptic search with the Smart Suction Cup on adversarial objects
• REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learning
• See, Hear, Explore: Curiosity via Audio-Visual Association
• GelBelt: A Vision-based Tactile Sensor for Continuous Sensing of Large Surfaces
• A Survey of Embodied Learning for Object-Centric Robotic Manipulation
• Cable Manipulation with a Tactile-Reactive Gripper
• MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning
• Planning to Repose Long and Heavy Objects Considering a Combination of Regrasp and Constrained Drooping
• Deep Tactile Experience: Estimating Tactile Sensor Output from Depth Sensor Data
• SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation
• Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data
• Event-Driven Visual-Tactile Sensing and Learning for Robots
• Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
• GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger
• A Modularized Design Approach for GelSight Family of Vision-based Tactile Sensors
• Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images
• Deep Robotic Prediction with hierarchical RGB-D Fusion
• Reaching, Grasping and Re-grasping: Learning Multimode Grasping Skills
• Contextual Reinforcement Learning of Visuo-tactile Multi-fingered Grasping Policies
• Stable Tool-Use with Flexible Musculoskeletal Hands by Learning the Predictive Model of Sensor State Transition
• ReSkin: versatile, replaceable, lasting tactile skins
• Grasping in the Dark: Zero-Shot Object Grasping Using Tactile Feedback
• Center-of-Mass-based Robust Grasp Planning for Unknown Objects Using Tactile-Visual Sensors
• TactileSGNet: A Spiking Graph Neural Network for Event-based Tactile Object Recognition
• Predicting Maximum Permitted Process Forces for Object Grasping and Manipulation Using a Deep Learning Regression Model
• Touch-based Curiosity for Sparse-Reward Tasks
• Deep Mixture Density Network for Probabilistic Object Detection
• Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception
• Optimal Deep Learning for Robot Touch
• Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard
• Towards Learning to Play Piano with Dexterous Hands and Touch
• 2.5D Image based Robotic Grasping