Ctrl-Z: Recovering from Instability in Reinforcement Learning

Abstract

When learning behavior, training data is often generated by the learner itself; this can result in unstable training dynamics, and this problem has particularly important applications in safety-sensitive real-world control tasks such as robotics. In this work, we propose a principled and model-agnostic approach to mitigate the issue of unstable learning dynamics by maintaining a history of a reinforcement learning agent over the course of training, and reverting to the parameters of a previous agent whenever performance significantly decreases. We develop techniques for evaluating this performance through statistical hypothesis testing of continued improvement, and evaluate them on a standard suite of challenging benchmark tasks involving continuous control of simulated robots. We show improvements over state-of-the-art reinforcement learning algorithms in performance and robustness to hyperparameters, outperforming DDPG in 5 out of 6 evaluation environments and showing no decrease in performance with TD3, which is known to be relatively stable. In this way, our approach takes an important step towards increasing data efficiency and stability in training for real-world robotic applications.

Submitted to ICRA2020, under review

References in corpus (10)

• Continuous control with deep reinforcement learning
• Overcoming catastrophic forgetting in neural networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Fully Distributed Multi-Robot Collision Avoidance via Deep Reinforcement Learning for Safe and Efficient Navigation in Complex Scenarios
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning

Cited by in corpus (4)

• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
• Goal-constrained Sparse Reinforcement Learning for End-to-End Driving
• The Quantum Cartpole: A benchmark environment for non-linear reinforcement learning
• On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration