Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning

Abstract

Robots must know how to be gentle when they need to interact with fragile objects, or when the robot itself is prone to wear and tear. We propose an approach that enables deep reinforcement learning to train policies that are gentle, both during exploration and task execution. In a reward-based learning environment, a natural approach involves augmenting the (task) reward with a penalty for non-gentleness, which can be defined as excessive impact force. However, augmenting with only this penalty impairs learning: policies get stuck in a local optimum which avoids all contact with the environment. Prior research has shown that combining auxiliary tasks or intrinsic rewards can be beneficial for stabilizing and accelerating learning in sparse-reward domains, and indeed we find that introducing a surprise-based intrinsic reward does avoid the no-contact failure case. However, we show that a simple dynamics-based surprise is not as effective as penalty-based surprise. Penalty-based surprise, based on predicting forceful contacts, has a further benefit: it encourages exploration which is contact-rich yet gentle. We demonstrate the effectiveness of the approach using a complex, tendon-powered robot hand with tactile sensors. Videos are available at http://sites.google.com/view/gentlemanipulation.

References in corpus (8)

• VIME: Variational Information Maximizing Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Safe Exploration in Continuous Action Spaces
• Reward Constrained Policy Optimization
• Constrained Policy Optimization
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Cited by in corpus (12)

• Making AI Intelligible: Philosophical Foundations
• A survey on intrinsic motivation in reinforcement learning
• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• Ctrl-Z: Recovering from Instability in Reinforcement Learning
• Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration
• Intrinsic Motivation for Encouraging Synergistic Behavior
• Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
• DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation
• Touch-based Curiosity for Sparse-Reward Tasks
• Fixed $\beta$-VAE Encoding for Curious Exploration in Complex 3D Environments
• ACDER: Augmented Curiosity-Driven Experience Replay
• Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard