Making Sense of Reinforcement Learning and Probabilistic Inference

Abstract

Reinforcement learning (RL) combines a control problem with statistical estimation: The system dynamics are not known to the agent, but can be learned through experience. A recent line of research casts `RL as inference' and suggests a particular framework to generalize the RL problem as probabilistic inference. Our paper surfaces a key shortcoming in that approach, and clarifies the sense in which RL can be coherently cast as an inference problem. In particular, an RL agent must consider the effects of its actions upon future rewards and observations: The exploration-exploitation tradeoff. In all but the most simple settings, the resulting inference is computationally intractable so that practical RL algorithms must resort to approximation. We demonstrate that the popular `RL as inference' approximation can perform poorly in even very basic problems. However, we show that with a small modification the framework does yield algorithms that can provably perform well, and we show that the resulting algorithm is equivalent to the recently proposed K-learning, which we further connect with Thompson sampling.

arXiv commentICLR 2020

References in corpus (11)

• Playing Atari with Deep Reinforcement Learning
• Weight Uncertainty in Neural Networks
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Optimal control as a graphical model inference problem
• Maximum a Posteriori Policy Optimisation
• Deep Exploration via Randomized Value Functions
• Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
• On Lower Bounds for Regret in Reinforcement Learning
• Behaviour Suite for Reinforcement Learning
• Variational Bayesian Reinforcement Learning with Regret Bounds

Cited by in corpus (7)

• On the Global Convergence Rates of Softmax Policy Gradient Methods
• The Two Kinds of Free Energy and the Bayesian Revolution
• Variational Bayesian Reinforcement Learning with Regret Bounds
• Reward is enough for convex MDPs
• Matrix games with bandit feedback
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Towards an Understanding of Default Policies in Multitask Policy Optimization