Optimistic Exploration even with a Pessimistic Initialisation

Abstract

Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algorithms do not use optimistic initialisation despite taking inspiration from these provably efficient tabular algorithms. In particular, in scenarios with only positive rewards, Q-values are initialised at their lowest possible values due to commonly used network initialisation schemes, a pessimistic initialisation. Merely initialising the network to output optimistic Q-values is not enough, since we cannot ensure that they remain optimistic for novel state-action pairs, which is crucial for exploration. We propose a simple count-based augmentation to pessimistically initialised Q-values that separates the source of optimism from the neural network. We show that this scheme is provably efficient in the tabular setting and extend it to the deep RL setting. Our algorithm, Optimistic Pessimistically Initialised Q-Learning (OPIQ), augments the Q-value estimates of a DQN-based agent with count-derived bonuses to ensure optimism during both action selection and bootstrapping. We show that OPIQ outperforms non-optimistic DQN variants that utilise a pseudocount-based intrinsic motivation in hard exploration tasks, and that it predicts optimistic estimates for novel state-action pairs.

Published as a conference paper at ICLR 2020

References in corpus (12)

• Conditional Image Generation with PixelCNN Decoders
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning Latent Dynamics for Planning from Pixels
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Deep Exploration via Randomized Value Functions
• Model-Based Active Exploration
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDP
• Approximate Exploration through State Abstraction
• Directed Exploration in PAC Model-Free Reinforcement Learning

Cited by in corpus (6)

• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• A Survey of Exploration Methods in Reinforcement Learning
• No-Regret Reinforcement Learning with Heavy-Tailed Rewards
• Towards Tractable Optimism in Model-Based Reinforcement Learning