The Two Kinds of Free Energy and the Bayesian Revolution

Abstract

The concept of free energy has its origins in 19th century thermodynamics, but has recently found its way into the behavioral and neural sciences, where it has been promoted for its wide applicability and has even been suggested as a fundamental principle of understanding intelligent behavior and brain function. We argue that there are essentially two different notions of free energy in current models of intelligent agency, that can both be considered as applications of Bayesian inference to the problem of action selection: one that appears when trading off accuracy and uncertainty based on a general maximum entropy principle, and one that formulates action selection in terms of minimizing an error measure that quantifies deviations of beliefs and policies from given reference models. The first approach provides a normative rule for action selection in the face of model uncertainty or when information processing capabilities are limited. The second approach directly aims to formulate the action selection problem as an inference problem in the context of Bayesian brain theories, also known as Active Inference in the literature. We elucidate the main ideas and discuss critical technical and conceptual issues revolving around these two notions of free energy that both claim to apply at all levels of decision-making, from the high-level deliberation of reasoning down to the low-level information processing of perception.

References in corpus (15)

• Asynchronous Methods for Deep Reinforcement Learning
• Expectation Propagation for approximate Bayesian inference
• Decision-Theoretic Planning: Structural Assumptions and Computational Leverage
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Optimal control as a graphical model inference problem
• Thermodynamics as a theory of decision-making with information processing costs
• Provably Bounded-Optimal Agents
• Stochastic thermodynamics of computation
• Information theoretic approach to interactive learning
• Some interesting observations on the free energy principle
• Bounded rational decision-making from elementary computations that reduce uncertainty
• Generalized Thompson Sampling for Sequential Decision-Making and Causal Inference
• The Efficiency of Human Cognition Reflects Planned Information Processing
• Making Sense of Reinforcement Learning and Probabilistic Inference

Cited by in corpus (6)

• Top influencers can be identified universally by combining classical centralities
• A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback
• Representing preorders with injective monotones
• Thermodynamic fluctuation theorems govern human sensorimotor learning
• Why collective behaviours self-organise to criticality: A primer on information-theoretic and thermodynamic utility measures
• Jarzyski's equality and Crooks' fluctuation theorem for general Markov chains with application to decision-making systems