Deep Active Inference

Abstract

This work combines the free energy principle from cognitive neuroscience and the ensuing active inference dynamics with recent advances in variational inference in deep generative models, and evolution strategies to introduce the "deep active inference" agent. This agent minimises a variational free energy bound on the average surprise of its sensations, which is motivated by a homeostatic argument. It does so by optimising the parameters of a generative latent variable model of its sensory inputs, together with a variational density approximating the posterior distribution over the latent variables, given its observations, and by acting on its environment to actively sample input that is likely under this generative model. The internal dynamics of the agent are implemented using deep and recurrent neural networks, as used in machine learning, making the deep active inference agent a scalable and very flexible class of active inference agent. Using the mountain car problem, we show how goal directed behaviour can be implemented by defining appropriate priors on the latent states in the agent's model. Furthermore, we show that the deep active inference agent can learn a generative model of the environment, which can be sampled from to understand the agent's beliefs about the environment and its interaction therewith.

References in corpus (9)

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Unsupervised Image-to-Image Translation Networks
• Universal Darwinism as a process of Bayesian inference
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Improving Variational Auto-Encoders using Householder Flow
• Variational Inference using Implicit Distributions
• Learning to Act by Predicting the Future
• An active inference implementation of phototaxis

Cited by in corpus (29)

• The free energy principle made simpler but not too simple
• Active inference: demystified and compared
• Reinforcement Learning through Active Inference
• An empirical evaluation of active inference in multi-armed bandits
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Neural dynamics under active inference: plausibility and efficiency of information processing
• Deep active inference agents using Monte-Carlo methods
• Interactive inference: a multi-agent model of cooperative joint actions
• Deep Active Inference for Partially Observable MDPs
• Bayesian Mechanics for Stationary Processes
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• Exploration and preference satisfaction trade-off in reward-free learning
• Active Inference and Epistemic Value in Graphical Models
• Deep Active Inference for Autonomous Robot Navigation
• Application of the Free Energy Principle to Estimation and Control
• Chance-Constrained Active Inference
• Active Inference Tree Search in Large POMDPs
• Active Inference or Control as Inference? A Unifying View
• On Predictive planning and counterfactual learning in active inference
• Cyber Kittens, or Some First Steps Towards Categorical Cybernetics
• Online reinforcement learning with sparse rewards through an active inference capsule
• Goal-Directed Planning by Reinforcement Learning and Active Inference
• Reinforced Imitation Learning by Free Energy Principle
• pymdp: A Python library for active inference in discrete state spaces
• Realising Active Inference in Variational Message Passing: the Outcome-blind Certainty Seeker
• Reactive Environments for Active Inference Agents with RxEnvironments.jl
• Episodic Memory for Learning Subjective-Timescale Models
• Hybrid Active Inference
• Explore and Control with Adversarial Surprise