Smooth Exploration for Robotic Reinforcement Learning
arXiv:2005.05719
Abstract
Reinforcement learning (RL) enables robots to learn skills from interactions with the real world. In practice, the unstructured step-based exploration used in Deep RL -- often very successful in simulation -- leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot. We address these issues by adapting state-dependent exploration (SDE) to current Deep RL algorithms. To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car. The noise sampling interval of gSDE permits to have a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance. The code is available at https://github.com/DLR-RM/stable-baselines3.
arXiv commentCode: https://github.com/DLR-RM/stable-baselines3/ Training scripts: https://github.com/DLR-RM/rl-baselines3-zoo/
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Soft Actor-Critic Algorithms and Applications
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Learning agile and dynamic motor skills for legged robots
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- Noisy Networks for Exploration
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- Parameter Space Noise for Exploration
- Learning Dexterous In-Hand Manipulation
- Unifying Count-Based Exploration and Intrinsic Motivation
- Simple random search provides a competitive approach to reinforcement learning
- CEM-RL: Combining evolutionary and gradient-based methods for policy search
- Learning to Walk in the Real World with Minimal Human Effort
- GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
- Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics
- Time Limits in Reinforcement Learning