Principled Exploration via Optimistic Bootstrapping and Backward Induction

Abstract

One principled approach for provably efficient exploration is incorporating the upper confidence bound (UCB) into the value function as a bonus. However, UCB is specified to deal with linear and tabular settings and is incompatible with Deep Reinforcement Learning (DRL). In this paper, we propose a principled exploration method for DRL through Optimistic Bootstrapping and Backward Induction (OB2I). OB2I constructs a general-purpose UCB-bonus through non-parametric bootstrap in DRL. The UCB-bonus estimates the epistemic uncertainty of state-action pairs for optimistic exploration. We build theoretical connections between the proposed UCB-bonus and the LSVI-UCB in a linear setting. We propagate future uncertainty in a time-consistent manner through episodic backward update, which exploits the theoretical advantage and empirically improves the sample-efficiency. Our experiments in the MNIST maze and Atari suite suggest that OB2I outperforms several state-of-the-art exploration approaches.

arXiv commentICML 2021

References in corpus (17)

• Continuous control with deep reinforcement learning
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Deep Exploration via Randomized Value Functions
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Never Give Up: Learning Directed Exploration Strategies
• UCB Exploration via Q-Ensembles
• Optimism in Reinforcement Learning with Generalized Linear Function Approximation
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Better Exploration with Optimistic Actor-Critic
• EMI: Exploration with Mutual Information
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• Optimistic Exploration even with a Pessimistic Initialisation
• Optimistic Policy Optimization with Bandit Feedback
• Information-Directed Exploration for Deep Reinforcement Learning
• Ready Policy One: World Building Through Active Learning

Cited by in corpus (2)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning