SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments

Abstract

Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche. We propose that such a struggle to achieve and preserve order might offer a principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing reinforcement learning (SMiRL). SMiRL alternates between learning a density model to evaluate the surprise of a stimulus, and improving the policy to seek more predictable stimuli. The policy seeks out stable and repeatable situations that counteract the environment's prevailing sources of entropy. This might include avoiding other hostile agents, or finding a stable, balanced pose for a bipedal robot in the face of disturbance forces. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, control a humanoid to avoid falls, and navigate to escape enemies in a maze without any task-specific reward supervision. We further show that SMiRL can be used together with standard task rewards to accelerate reward-driven learning.

arXiv commentICLR 2021

References in corpus (18)

• Auto-Encoding Variational Bayes
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• VIME: Variational Information Maximizing Exploration
• Emergent Tool Use From Multi-Agent Autocurricula
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Exploration by Random Network Distillation
• Emergent Complexity via Multi-Agent Competition
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Deep Active Inference
• Provably Efficient Maximum Entropy Exploration
• Generative Adversarial Imitation from Observation
• Efficient Exploration via State Marginal Matching
• Reinforcement Learning through Active Inference
• Imitating Latent Policies from Observation
• Active World Model Learning with Progress Curiosity

Cited by in corpus (6)

• Action and Perception as Divergence Minimization
• Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study
• Grounding Artificial Intelligence in the Origins of Human Behavior
• Adversarial Intrinsic Motivation for Reinforcement Learning
• Adapting Surprise Minimizing Reinforcement Learning Techniques for Transactive Control
• Explore and Control with Adversarial Surprise