Action and Perception as Divergence Minimization

Abstract

To learn directed behaviors in complex environments, intelligent agents need to optimize objective functions. Various objectives are known for designing artificial agents, including task rewards and intrinsic motivation. However, it is unclear how the known objectives relate to each other, which objectives remain yet to be discovered, and which objectives better describe the behavior of humans. We introduce the Action Perception Divergence (APD), an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general objectives. While the narrow objectives correspond to domain-specific rewards as typical in reinforcement learning, the general objectives maximize information with the environment through latent variable models of input sequences. Intuitively, these agents use perception to align their beliefs with the world and use actions to align the world with their beliefs. They infer representations that are informative of past inputs, explore future inputs that are informative of their representations, and select actions or skills that maximally influence future inputs. This explains a wide range of unsupervised objectives from a single principle, including representation learning, information gain, empowerment, and skill discovery. Our findings suggest leveraging powerful world models for unsupervised exploration as a path toward highly adaptive agents that seek out large niches in their environments, rendering task rewards optional.

Website: https://danijar.com/apd

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Weight Uncertainty in Neural Networks
• A free energy principle for a particular physics
• Variational Intrinsic Control
• Functional Variational Bayesian Neural Networks
• Self-Supervised Visual Planning with Temporal Skip Connections
• Planning to Explore via Self-Supervised World Models
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• A Divergence Minimization Perspective on Imitation Learning Methods
• Approximate Inference and Stochastic Optimal Control
• World Discovery Models
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network

Cited by in corpus (8)

• The whole brain architecture approach: Accelerating the development of artificial general intelligence by referring to the brain
• Evaluating Agents without Rewards
• Active Inference and Epistemic Value in Graphical Models
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Online reinforcement learning with sparse rewards through an active inference capsule
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning