Simplifying Deep Reinforcement Learning via Self-Supervision
arXiv:2106.05526
Abstract
Supervised regression to demonstrations has been demonstrated to be a stable way to train deep policy networks. We are motivated to study how we can take full advantage of supervised loss functions for stably training deep reinforcement learning agents. This is a challenging task because it is unclear how the training data could be collected to enable policy improvement. In this work, we propose Self-Supervised Reinforcement Learning (SSRL), a simple algorithm that optimizes policies with purely supervised losses. We demonstrate that, without policy gradient or value estimation, an iterative procedure of ``labeling" data and supervised regression is sufficient to drive stable policy improvement. By selecting and imitating trajectories with high episodic rewards, SSRL is surprisingly competitive to contemporary algorithms with more stable performance and less running time, showing the potential of solving reinforcement learning with supervised learning techniques. The code is available at https://github.com/daochenzha/SSRL
References in corpus (14)
- Trust Region Policy Optimization
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Deep Reinforcement Learning that Matters
- Implicit Quantile Networks for Distributional Reinforcement Learning
- Maximum a Posteriori Policy Optimisation
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- A Deeper Look at Experience Replay
- Dopamine: A Research Framework for Deep Reinforcement Learning
- Policy Distillation
- RLCard: A Toolkit for Reinforcement Learning in Card Games
- Learning Self-Imitating Diverse Policies
- Training Agents using Upside-Down Reinforcement Learning
- AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning