When should agents explore?
arXiv:2108.11811
Abstract
Exploration remains a central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of switching between modes. This paper presents an initial study of mode-switching, non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter-tuning burden. Finally, we report a promising and detailed analysis on Atari, using two-mode exploration and switching at sub-episodic time-scales.
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Dueling Network Architectures for Deep Reinforcement Learning
- VIME: Variational Information Maximizing Exploration
- Distributed Prioritized Experience Replay
- Count-Based Exploration with Neural Density Models
- Exploration by Random Network Distillation
- Agent57: Outperforming the Atari Human Benchmark
- Go-Explore: a New Approach for Hard-Exploration Problems
- Variational Intrinsic Control
- Lévy flights in human behavior and cognition
- Never Give Up: Learning Directed Exploration Strategies
- Adaptive Skills, Adaptive Partitions (ASAP)
- Tactical Optimism and Pessimism for Deep Reinforcement Learning
- Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
- Discovering Options for Exploration by Minimizing Cover Time
- Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
- Adapting Behaviour for Learning Progress
- Temporal Difference Uncertainties as a Signal for Exploration
- Return-based Scaling: Yet Another Normalisation Trick for Deep RL
- Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments