UCB Exploration via Q-Ensembles

Abstract

We show how an ensemble of $Q^{\ast }$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.

References in corpus (4)

• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Model-Based Bayesian Exploration
• (More) Efficient Reinforcement Learning via Posterior Sampling
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Cited by in corpus (25)

• Large-Scale Study of Curiosity-Driven Learning
• Exploration by Random Network Distillation
• Evolved Policy Gradients
• Learning Montezuma's Revenge from a Single Demonstration
• Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reduction
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• Continuous Doubly Constrained Batch Reinforcement Learning
• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• Differentiable Linear Bandit Algorithm
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Learning to Map for Active Semantic Goal Navigation
• Learning more skills through optimistic exploration
• Policy Optimization with Model-based Explorations
• Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control
• Variational Bayes: A report on approaches and applications
• REMAX: Relational Representation for Multi-Agent Exploration
• Reannealing of Decaying Exploration Based On Heuristic Measure in Deep Q-Network
• Bayesian Distributional Policy Gradients
• DQ Scheduler: Deep Reinforcement Learning Based Controller Synchronization in Distributed SDN
• Combining Model-Free Q-Ensembles and Model-Based Approaches for Informed Exploration