Action and Perception as Divergence Minimization

Abstract

To learn directed behaviors in complex environments, intelligent agents need to optimize objective functions. Various objectives are known for designing artificial agents, including task rewards and intrinsic motivation. However, it is unclear how the known objectives relate to each other, which objectives remain yet to be discovered, and which objectives better describe the behavior of humans. We introduce the Action Perception Divergence (APD), an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general objectives. While the narrow objectives correspond to domain-specific rewards as typical in reinforcement learning, the general objectives maximize information with the environment through latent variable models of input sequences. Intuitively, these agents use perception to align their beliefs with the world and use actions to align the world with their beliefs. They infer representations that are informative of past inputs, explore future inputs that are informative of their representations, and select actions or skills that maximally influence future inputs. This explains a wide range of unsupervised objectives from a single principle, including representation learning, information gain, empowerment, and skill discovery. Our findings suggest leveraging powerful world models for unsupervised exploration as a path toward highly adaptive agents that seek out large niches in their environments, rendering task rewards optional.

arXiv commentWebsite: https://danijar.com/apd

References in corpus (51)

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Weight Uncertainty in Neural Networks
• VIME: Variational Information Maximizing Exploration
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Optimal control as a graphical model inference problem
• On Mutual Information Maximization for Representation Learning
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Exploration by Random Network Distillation
• A free energy principle for a particular physics
• Averaging Weights Leads to Wider Optima and Better Generalization
• Episodic Curiosity through Reachability
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Variational Intrinsic Control
• Deep Kalman Filters
• Dream to Control: Learning Behaviors by Latent Imagination
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Functional Variational Bayesian Neural Networks
• Self-Supervised Visual Planning with Temporal Skip Connections
• The Conditional Entropy Bottleneck
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Efficient Exploration via State Marginal Matching
• Variational Option Discovery Algorithms
• Bayesian Layers: A Module for Neural Network Uncertainty
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• Dynamics-Aware Unsupervised Discovery of Skills
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Taming the Noise in Reinforcement Learning via Soft Updates
• Planning to Explore via Self-Supervised World Models
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Neural Predictive Belief Representations
• Semi-Amortized Variational Autoencoders
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• A unified strategy for implementing curiosity and empowerment driven reinforcement learning
• A Divergence Minimization Perspective on Imitation Learning Methods
• TherML: Thermodynamics of Machine Learning
• Approximate Inference and Stochastic Optimal Control
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• World Discovery Models
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network

Cited by in corpus (10)

• The whole brain architecture approach: Accelerating the development of artificial general intelligence by referring to the brain
• Evaluating Agents without Rewards
• Active Inference and Epistemic Value in Graphical Models
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Online reinforcement learning with sparse rewards through an active inference capsule
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Joint Perception and Control as Inference with an Object-based Implementation
• Learning Transferable Concepts in Deep Reinforcement Learning
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning