Touch-based Curiosity for Sparse-Reward Tasks

Abstract

Robots in many real-world settings have access to force/torque sensors in their gripper and tactile sensing is often necessary in tasks that involve contact-rich motion. In this work, we leverage surprise from mismatches in touch feedback to guide exploration in hard sparse-reward reinforcement learning tasks. Our approach, Touch-based Curiosity (ToC), learns what visible objects interactions are supposed to "feel" like. We encourage exploration by rewarding interactions where the expectation and the experience don't match. In our proposed method, an initial task-independent exploration phase is followed by an on-task learning phase, in which the original interactions are relabeled with on-task rewards. We test our approach on a range of touch-intensive robot arm tasks (e.g. pushing objects, opening doors), which we also release as part of this work. Across multiple experiments in a simulated setting, we demonstrate that our method is able to learn these difficult tasks through sparse reward and curiosity alone. We compare our cross-modal approach to single-modality (touch- or vision-only) approaches as well as other curiosity-based methods and find that our method performs better and is more sample-efficient.

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Empirical Evaluation of Rectified Activations in Convolutional Network
• More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Hindsight Experience Replay
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Episodic Curiosity through Reachability
• Variational Intrinsic Control
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• 3D Shape Perception from Monocular Vision, Touch, and Shape Priors
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning