Reinforcement Learning as Iterative and Amortised Inference

Abstract

There are several ways to categorise reinforcement learning (RL) algorithms, such as either model-based or model-free, policy-based or planning-based, on-policy or off-policy, and online or offline. Broad classification schemes such as these help provide a unified perspective on disparate techniques and can contextualise and guide the development of new algorithms. In this paper, we utilise the control as inference framework to outline a novel classification scheme based on amortised and iterative inference. We demonstrate that a wide range of algorithms can be classified in this manner providing a fresh perspective and highlighting a range of existing similarities. Moreover, we show that taking this perspective allows us to identify parts of the algorithmic design space which have been relatively unexplored, suggesting new routes to innovative RL algorithms.

initial upload; 05-07-20 -- updated with minor corrections

References in corpus (14)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Adaptive importance sampling for control and inference
• Universal Planning Networks
• Black Box Variational Inference
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Iterative Amortized Inference
• Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference