VIREL: A Variational Inference Framework for Reinforcement Learning

Abstract

Applying probabilistic models to reinforcement learning (RL) enables the application of powerful optimisation tools such as variational inference to RL. However, existing inference frameworks and their algorithms pose significant challenges for learning optimal policies, e.g., the absence of mode capturing behaviour in pseudo-likelihood methods and difficulties learning deterministic policies in maximum entropy RL based approaches. We propose VIREL, a novel, theoretically grounded probabilistic inference framework for RL that utilises a parametrised action-value function to summarise future dynamics of the underlying MDP. This gives VIREL a mode-seeking form of KL divergence, the ability to learn deterministic optimal polices naturally from inference and the ability to optimise value functions and policies in separate, iterative steps. In applying variational expectation-maximisation to VIREL we thus show that the actor-critic algorithm can be reduced to expectation-maximisation, with policy improvement equivalent to an E-step and policy evaluation to an M-step. We then derive a family of actor-critic methods from VIREL, including a scheme for adaptive exploration. Finally, we demonstrate that actor-critic algorithms from this family outperform state-of-the-art methods based on soft value functions in several domains.

References in corpus (16)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Continuous Control Policies by Stochastic Value Gradients
• Maximum a Posteriori Policy Optimisation
• Equivalence Between Policy Gradients and Soft Q-Learning
• Gradient Estimation Using Stochastic Computation Graphs
• Policy Iteration for Factored MDPs
• A Theoretical Analysis of Deep Q-Learning
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• A Theory of Regularized Markov Decision Processes
• MAVEN: Multi-Agent Variational Exploration
• Expected Policy Gradients for Reinforcement Learning
• Approximate Inference and Stochastic Optimal Control
• Symmetry Learning for Function Approximation in Reinforcement Learning
• Fourier Policy Gradients

Cited by in corpus (10)

• Deep active inference agents using Monte-Carlo methods
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Outcome-Driven Reinforcement Learning via Variational Inference
• Hierarchical Variational Imitation Learning of Control Programs
• Online Variational Filtering and Parameter Learning
• Self-Paced Deep Reinforcement Learning
• Learning Policies through Quantile Regression
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning