Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control

Abstract

The exploration-exploitation trade-off is central to the description of adaptive behaviour in fields ranging from machine learning, to biology, to economics. While many approaches have been taken, one approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' which is often implemented in terms of maximizing the agents information gain about the world -- an approach which has been widely studied in machine learning and cognitive science. In this paper we mathematically investigate the nature and meaning of such approaches and demonstrate that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives we call divergence objectives. We propose a dichotomy in the objective functions underlying adaptive behaviour between \emph{evidence} objectives, which correspond to well-known reward or utility maximizing objectives in the literature, and \emph{divergence} objectives which instead seek to minimize the divergence between the agent's expected and desired futures, and argue that this new class of divergence objectives could form the mathematical foundation for a much richer understanding of the exploratory components of adaptive and intelligent action, beyond simply greedy utility maximization.

arXiv comment11-03-21 initial upload. 14-03-21 fix Charnov citation. 16-03-21 another fix. 25-06-21 more fixes plus numerical simulations. 30-06-21 minor fixes; 12/11/21 maths typo fix; 24/11/21 minor maths fixes

References in corpus (19)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• VIME: Variational Information Maximizing Exploration
• Recurrent World Models Facilitate Policy Evolution
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Path integrals and symmetry breaking for optimal control theory
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits
• Variational Intrinsic Control
• Efficient Exploration via State Marginal Matching
• Model-Based Active Exploration
• Reinforcement Learning through Active Inference
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Deep active inference agents using Monte-Carlo methods
• Action and Perception as Divergence Minimization
• Approximate Inference and Stochastic Optimal Control
• Relative Variational Intrinsic Control
• Online reinforcement learning with sparse rewards through an active inference capsule

Cited by in corpus (1)

• Active Inference and Epistemic Value in Graphical Models