Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
arXiv:2006.07554
Abstract
Off-policy learning algorithms have been known to be sensitive to the choice of hyper-parameters. However, unlike near on-policy algorithms for which hyper-parameters could be optimized via e.g. meta-gradients, similar techniques could not be straightforwardly applied to off-policy learning. In this work, we propose a framework which entails the application of Evolutionary Strategies to online hyper-parameter tuning in off-policy learning. Our formulation draws close connections to meta-gradients and leverages the strengths of black-box optimization with relatively low-dimensional search spaces. We show that our method outperforms state-of-the-art off-policy learning baselines with static hyper-parameters and recent prior work over a wide range of continuous control benchmarks.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- DeepMind Control Suite
- Distributed Distributional Deterministic Policy Gradients
- Maximum a Posteriori Policy Optimisation
- Meta-Gradient Reinforcement Learning
- Simple random search provides a competitive approach to reinforcement learning
- Deep Reinforcement Learning and the Deadly Triad
- CEM-RL: Combining evolutionary and gradient-based methods for policy search
- Structured Evolution with Compact Architectures for Scalable Policy Optimization
- A Self-Tuning Actor-Critic Algorithm
- Fast Efficient Hyperparameter Tuning for Policy Gradients
- Adaptive Trade-Offs in Off-Policy Learning
- Adaptive Lambda Least-Squares Temporal Difference Learning