Self-Imitation Learning via Generalized Lower Bound Q-learning
arXiv:2006.07442
Abstract
Self-imitation learning motivated by lower-bound Q-learning is a novel and effective approach for off-policy learning. In this work, we propose a n-step lower bound which generalizes the original return-based lower-bound Q-learning, and introduce a new family of self-imitation learning algorithms. To provide a formal motivation for the potential performance gains provided by self-imitation learning, we show that n-step lower bound Q-learning achieves a trade-off between fixed point bias and contraction rate, drawing close connections to the popular uncorrected n-step Q-learning. We finally show that n-step lower bound Q-learning is a more robust alternative to return-based self-imitation learning and uncorrected n-step, over a wide range of continuous control benchmark tasks.
arXiv commentAccepted at NeurIPS (Neural Information Processing Systems) 2020, Vancouver, Canada. Code is available at https://github.com/robintyh1/nstep-sil
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Prioritized Experience Replay
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- DeepMind Control Suite
- Distributed Prioritized Experience Replay
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Distributed Distributional Deterministic Policy Gradients
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Model-Free Episodic Control
- Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
- Taming the Noise in Reinforcement Learning via Soft Updates
- Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
- Learning Self-Imitating Diverse Policies
- Multi-step Off-policy Learning Without Importance Sampling Ratios
- Adaptive Trade-Offs in Off-Policy Learning