Intrinsic Exploration as Multi-Objective RL
arXiv:2004.02380
Abstract
Intrinsic motivation enables reinforcement learning (RL) agents to explore when rewards are very sparse, where traditional exploration heuristics such as Boltzmann or e-greedy would typically fail. However, intrinsic exploration is generally handled in an ad-hoc manner, where exploration is not treated as a core objective of the learning process; this weak formulation leads to sub-optimal exploration performance. To overcome this problem, we propose a framework based on multi-objective RL where both exploration and exploitation are being optimized as separate objectives. This formulation brings the balance between exploration and exploitation at a policy level, resulting in advantages over traditional methods. This also allows for controlling exploration while learning, at no extra cost. Such strategies achieve a degree of control over agent exploration that was previously unattainable with classic or intrinsic rewards. We demonstrate scalability to continuous state-action spaces by presenting a method (EMU-Q) based on our framework, guiding exploration towards regions of higher value-function uncertainty. EMU-Q is experimentally shown to outperform classic exploration techniques and other intrinsic RL methods on a continuous control benchmark and on a robotic manipulator.
References in corpus (8)
- Trust Region Policy Optimization
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
- Algorithms for multi-armed bandit problems
- Generalization and Exploration via Randomized Value Functions
- Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels
- DORA The Explorer: Directed Outreaching Reinforcement Action-Selection
- On the Error of Random Fourier Features