Bayesian Reinforcement Learning: A Survey

Abstract

Bayesian methods for machine learning have been widely investigated, yielding principled methods for incorporating prior information into inference algorithms. In this survey, we provide an in-depth review of the role of Bayesian methods for the reinforcement learning (RL) paradigm. The major incentives for incorporating Bayesian reasoning in RL are: 1) it provides an elegant approach to action-selection (exploration/exploitation) as a function of the uncertainty in learning; and 2) it provides a machinery to incorporate prior knowledge into the algorithms. We first discuss models and methods for Bayesian inference in the simple single-step Bandit model. We then review the extensive recent literature on Bayesian methods for model-based RL, where prior information can be expressed on the parameters of the Markov model. We also present Bayesian methods for model-free RL, where priors are expressed over the value function or policy class. The objective of the paper is to provide a comprehensive survey on Bayesian RL algorithms and their theoretical and empirical properties.

References in corpus (16)

• Further Optimal Regret Bounds for Thompson Sampling
• Model-Based Bayesian Exploration
• (More) Efficient Reinforcement Learning via Posterior Sampling
• A Bayesian Sampling Approach to Exploration in Reinforcement Learning
• Experiments with Infinite-Horizon, Policy-Gradient Estimation
• Computing a classic index for finite-horizon bandits
• Bayesian multitask inverse reinforcement learning
• An Information-Theoretic Analysis of Thompson Sampling
• Model-Based Bayesian Reinforcement Learning in Large Structured Domains
• Bayesian policy gradient and actor-critic algorithms
• Near-Optimal BRL using Optimistic Local Transitions
• Thompson Sampling for Learning Parameterized Markov Decision Processes
• Benchmarking for Bayesian Reinforcement Learning
• A Greedy Approximation of Bayesian Reinforcement Learning with Probably Optimistic Transition Model
• PAC-Bayesian Policy Evaluation for Reinforcement Learning
• PAC-Bayesian Analysis of the Exploration-Exploitation Trade-off

Cited by in corpus (19)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Partially Observable Markov Decision Processes in Robotics: A Survey
• ReLU Networks as Surrogate Models in Mixed-Integer Linear Programs
• Active inference: demystified and compared
• Partially Observable Markov Decision Processes (POMDPs) and Robotics
• A cooperative game for automated learning of elasto-plasticity knowledge graphs and models with AI-guided experimentation
• An empirical evaluation of active inference in multi-armed bandits
• A Tutorial on Meta-Reinforcement Learning
• Quantum Reinforcement Learning: the Maze problem
• Personalized Optimization with User's Feedback
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Digital Twin-based 3D Map Management for Edge-assisted Device Pose Tracking in Mobile AR
• Dual Control of Exploration and Exploitation for Auto-Optimisation Control with Active Learning
• On the Benefit of Optimal Transport for Curriculum Reinforcement Learning
• Entropic Regularization of Markov Decision Processes
• Preparing for the Unexpected: Diversity Improves Planning Resilience in Evolutionary Algorithms
• RLOps: Development Life-cycle of Reinforcement Learning Aided Open RAN
• Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery