Towards Tractable Optimism in Model-Based Reinforcement Learning
arXiv:2006.11911
Abstract
The principle of optimism in the face of uncertainty is prevalent throughout sequential decision making problems such as multi-armed bandits and reinforcement learning (RL). To be successful, an optimistic RL algorithm must over-estimate the true value function (optimism) but not by so much that it is inaccurate (estimation error). In the tabular setting, many state-of-the-art methods produce the required optimism through approaches which are intractable when scaling to deep RL. We re-interpret these scalable optimistic model-based algorithms as solving a tractable noise augmented MDP. This formulation achieves a competitive regret bound: when augmenting using Gaussian noise, where is the total number of environment steps. We also explore how this trade-off changes in the deep RL setting, where we show empirically that estimation error is significantly more troublesome. However, we also show that if this error is reduced, optimistic model-based RL algorithms can match state-of-the-art performance in continuous control problems.
arXiv commentPresented as a conference paper at UAI 2021
References in corpus (9)
- Large-Scale Study of Curiosity-Driven Learning
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Better Exploration with Optimistic Actor-Critic
- Worst-Case Regret Bounds for Exploration via Randomized Value Functions
- Optimistic Exploration even with a Pessimistic Initialisation
- Near-optimal Optimistic Reinforcement Learning using Empirical Bernstein Inequalities
- New Insights into Bootstrapping for Bandits