Contrastive Active Inference

Abstract

Active inference is a unifying theory for perception and action resting upon the idea that the brain maintains an internal model of the world by minimizing free energy. From a behavioral perspective, active inference agents can be seen as self-evidencing beings that act to fulfill their optimistic predictions, namely preferred outcomes or goals. In contrast, reinforcement learning requires human-designed rewards to accomplish any desired outcome. Although active inference could provide a more natural self-supervised objective for control, its applicability has been limited because of the shortcomings in scaling the approach to complex environments. In this work, we propose a contrastive objective for active inference that strongly reduces the computational burden in learning the agent's generative model and planning future actions. Our method performs notably better than likelihood-based active inference in image-based tasks, while also being computationally cheaper and easier to train. We compare to reinforcement learning agents that have access to human-designed reward functions, showing that our approach closely matches their performance. Finally, we also show that contrastive methods perform significantly better in the case of distractors in the environment and that our method is able to generalize goals to variations in the background. Website and code: https://contrastive-aif.github.io/

Accepted as a conference paper at 35th Conference on Neural Information Processing Systems (NeurIPS 2021)

References in corpus (17)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• A Simple Framework for Contrastive Learning of Visual Representations
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Improved Baselines with Momentum Contrastive Learning
• Solving Rubik's Cube with a Robot Hand
• DeepMind Control Suite
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Agent57: Outperforming the Atari Human Benchmark
• On Variational Bounds of Mutual Information
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Reinforcement Learning through Active Inference
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Model-Augmented Actor-Critic: Backpropagating through Paths
• The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels