Adapting Behaviour for Learning Progress
arXiv:1912.06910
Abstract
Determining what experience to generate to best facilitate learning (i.e. exploration) is one of the distinguishing features and open challenges in reinforcement learning. The advent of distributed agents that interact with parallel instances of the environment has enabled larger scales and greater flexibility, but has not removed the need to tune exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning. We propose to dynamically adapt the data generation by using a non-stationary multi-armed bandit to optimize a proxy of the learning progress. The data distribution is controlled by modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without significant overhead. The adaptation speed of the bandit can be increased by exploiting the factored modulation structure. We demonstrate on a suite of Atari 2600 games how this unified approach produces results comparable to per-task tuning at a fraction of the cost.
References in corpus (10)
- Prioritized Experience Replay
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Distributed Prioritized Experience Replay
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Distributional Reinforcement Learning with Quantile Regression
- Latent Space Policies for Hierarchical Reinforcement Learning
- Taming Non-stationary Bandits: A Bayesian Approach
- Mix&Match - Agent Curricula for Reinforcement Learning
- On Inductive Biases in Deep Reinforcement Learning
- Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
Cited by in corpus (7)
- Agent57: Outperforming the Atari Human Benchmark
- Effective Diversity in Population Based Reinforcement Learning
- A Self-Tuning Actor-Critic Algorithm
- Tactical Optimism and Pessimism for Deep Reinforcement Learning
- Temporal Difference Uncertainties as a Signal for Exploration
- Return-based Scaling: Yet Another Normalisation Trick for Deep RL
- When should agents explore?