Model-Based Bayesian Exploration

Abstract

Reinforcement learning systems are often concerned with balancing exploration of untested actions against exploitation of actions that are known to be good. The benefit of exploration can be estimated using the classical notion of Value of Information - the expected improvement in future decision quality arising from the information acquired by exploration. Estimating this quantity requires an assessment of the agent's uncertainty about its current value estimates for states. In this paper we investigate ways of representing and reasoning about this uncertainty in algorithms where the system attempts to learn a model of its environment. We explicitly represent uncertainty about the parameters of the model and build probability distributions over Q-values based on these. These distributions are used to compute a myopic approximation to the value of information for each action and hence to select the action that best balances exploration and exploitation.

Appears in Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence (UAI1999)

References in corpus (1)

• Stochastic Simulation Algorithms for Dynamic Probabilistic Networks

Cited by in corpus (30)

• An Algorithmic Perspective on Imitation Learning
• Benchmarking Model-Based Reinforcement Learning
• Bayesian Reinforcement Learning: A Survey
• Accelerating Reinforcement Learning through Implicit Imitation
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Evolved Policy Gradients
• Learning Montezuma's Revenge from a Single Demonstration
• UCB Exploration via Q-Ensembles
• Belief and Truth in Hypothesised Behaviours
• Model-Based Bayesian Reinforcement Learning in Large Structured Domains
• Bayesian policy gradient and actor-critic algorithms
• Dynamic Decision Making for Graphical Models Applied to Oil Exploration
• Thompson Sampling for Learning Parameterized Markov Decision Processes
• Guided Policy Exploration for Markov Decision Processes using an Uncertainty-Based Value-of-Information Criterion
• Monte Carlo Bayesian Reinforcement Learning
• Efficient exploration with Double Uncertain Value Networks
• Benchmarking for Bayesian Reinforcement Learning
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Optimal Reinforcement Learning for Gaussian Systems
• Assumed Density Filtering Q-learning
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• A Survey of Exploration Methods in Reinforcement Learning
• Autonomous Thermalling as a Partially Observable Markov Decision Process (Extended Version)
• A Minimum Relative Entropy Principle for Learning and Acting
• Hyper-parameter Tuning under a Budget Constraint
• MDPs with Unawareness in Robotics
• Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization
• Dual Control for Approximate Bayesian Reinforcement Learning
• Bayesian Policy Reuse
• Inferential Induction: A Novel Framework for Bayesian Reinforcement Learning