A Bayesian Sampling Approach to Exploration in Reinforcement Learning

Abstract

We present a modular approach to reinforcement learning that uses a Bayesian representation of the uncertainty over models. The approach, BOSS (Best of Sampled Set), drives exploration by sampling multiple models from the posterior and selecting actions optimistically. It extends previous work by providing a rule for deciding when to resample and how to combine the models. We show that our algorithm achieves nearoptimal reward with high probability with a sample complexity that is low relative to the speed at which the posterior distribution converges during learning. We demonstrate that BOSS performs quite favorably compared to state-of-the-art reinforcement-learning approaches and illustrate its flexibility by pairing it with a non-parametric model that generalizes across states.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

References in corpus (2)

• Church: a language for generative models
• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation

Cited by in corpus (31)

• Bayesian Reinforcement Learning: A Survey
• Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
• Reinforcement learning for port-Hamiltonian systems
• Learning is planning: near Bayes-optimal reinforcement learning via Monte-Carlo tree search
• Bayesian policy gradient and actor-critic algorithms
• A Tutorial on Ultra-Reliable and Low-Latency Communications in 6G: Integrating Domain Knowledge into Deep Learning
• Near-Optimal BRL using Optimistic Local Transitions
• Monte Carlo Bayesian Reinforcement Learning
• Cover Tree Bayesian Reinforcement Learning
• Worst-Case Regret Bounds for Exploration via Randomized Value Functions
• Surprising Negative Results for Generative Adversarial Tree Search
• Learning in embodied action-perception loops through exploration
• Benchmarking for Bayesian Reinforcement Learning
• Optimal Reinforcement Learning for Gaussian Systems
• Variance-Based Rewards for Approximate Bayesian Reinforcement Learning
• Learning Awareness Models
• Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
• Apprenticeship Learning for Model Parameters of Partially Observable Environments
• A Survey of Exploration Methods in Reinforcement Learning
• Robust Bayesian reinforcement learning through tight lower bounds
• Complexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning
• Bayesian Reinforcement Learning via Deep, Sparse Sampling
• POMDP-lite for Robust Robot Planning under Uncertainty
• Better Optimism By Bayes: Adaptive Planning with Rich Models
• Posterior sampling for reinforcement learning: worst-case regret bounds
• Temporally-Extended ε-Greedy Exploration
• Sparse Reward Processes
• Biomanufacturing Harvest Optimization with Small Data
• Bayes-CPACE: PAC Optimal Exploration in Continuous Space Bayes-Adaptive Markov Decision Processes
• Reinforcement Learning in Reward-Mixing MDPs
• TSEB: More Efficient Thompson Sampling for Policy Learning