Reinforcement Learning through Active Inference

Abstract

The central tenet of reinforcement learning (RL) is that agents seek to maximize the sum of cumulative rewards. In contrast, active inference, an emerging framework within cognitive and computational neuroscience, proposes that agents act to maximize the evidence for a biased generative model. Here, we illustrate how ideas from active inference can augment traditional RL approaches by (i) furnishing an inherent balance of exploration and exploitation, and (ii) providing a more flexible conceptualization of reward. Inspired by active inference, we develop and implement a novel objective for decision making, which we term the free energy of the expected future. We demonstrate that the resulting algorithm successfully balances exploration and exploitation, simultaneously achieving robust performance on several challenging RL benchmarks with sparse, well-shaped, and no rewards.

References in corpus (21)

• Auto-Encoding Variational Bayes
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Weight Uncertainty in Neural Networks
• VIME: Variational Information Maximizing Exploration
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Deep Ensembles: A Loss Landscape Perspective
• Optimal control as a graphical model inference problem
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• A free energy principle for a particular physics
• Dream to Control: Learning Behaviors by Latent Imagination
• Deep Active Inference
• Efficient Exploration via State Marginal Matching
• Model-Based Active Exploration
• Deep Dynamics Models for Learning Dexterous Manipulation
• The Uncertainty Bellman Equation and Exploration
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments
• A unified strategy for implementing curiosity and empowerment driven reinforcement learning
• Variational Inference for Data-Efficient Model Learning in POMDPs

Cited by in corpus (14)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Neural dynamics under active inference: plausibility and efficiency of information processing
• Equilibrium in the Computing Continuum through Active Inference
• Predictive Coding: a Theoretical and Experimental Review
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Active Inference and Epistemic Value in Graphical Models
• Chance-Constrained Active Inference
• Active Inference or Control as Inference? A Unifying View
• Online reinforcement learning with sparse rewards through an active inference capsule
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• A Mathematical Walkthrough and Discussion of the Free Energy Principle
• Contrastive Active Inference