Variational Bayesian Reinforcement Learning with Regret Bounds

Abstract

In reinforcement learning the Q-values summarize the expected future rewards that the agent will attain. However, they cannot capture the epistemic uncertainty about those rewards. In this work we derive a new Bellman operator with associated fixed point we call the `knowledge values'. These K-values compress both the expected future rewards and the epistemic uncertainty into a single value, so that high uncertainty, high reward, or both, can yield high K-values. The key principle is to endow the agent with a risk-seeking utility function that is carefully tuned to balance exploration and exploitation. When the agent follows a Boltzmann policy over the K-values it yields a Bayes regret bound of $\tilde{O}(L\sqrt{SAT})$, where $L$ is the time horizon, $S$ is the total number of states, $A$ is the number of actions, and $T$ is the number of elapsed timesteps. We show deep connections of this approach to the soft-max and maximum-entropy strands of research in reinforcement learning.

References in corpus (20)

• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Count-Based Exploration with Neural Density Models
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Thermodynamics as a theory of decision-making with information processing costs
• Unifying Count-Based Exploration and Intrinsic Motivation
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Deep Exploration via Randomized Value Functions
• Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
• Follow the Leader If You Can, Hedge If You Must
• Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
• Generalization and Exploration via Randomized Value Functions
• Information-Theoretic Bounded Rationality
• Near-Optimal BRL using Optimistic Local Transitions
• Making Sense of Reinforcement Learning and Probabilistic Inference
• Free Energy and the Generalized Optimality Equations for Sequential Decision Making
• A short variational proof of equivalence between policy gradients and soft Q learning
• Gaussian-Dirichlet Posterior Dominance in Sequential Learning

Cited by in corpus (8)

• Conservative Q-Learning for Offline Reinforcement Learning
• If MaxEnt RL is the Answer, What is the Question?
• Making Sense of Reinforcement Learning and Probabilistic Inference
• Reinforcement Learning, Bit by Bit
• Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spaces
• Reward is enough for convex MDPs
• Ranking Policy Gradient
• Matrix games with bandit feedback