Explore and Control with Adversarial Surprise

Abstract

Unsupervised reinforcement learning (RL) studies how to leverage environment statistics to learn useful behaviors without the cost of reward engineering. However, a central challenge in unsupervised RL is to extract behaviors that meaningfully affect the world and cover the range of possible outcomes, without getting distracted by inherently unpredictable, uncontrollable, and stochastic elements in the environment. To this end, we propose an unsupervised RL method designed for high-dimensional, stochastic environments based on an adversarial game between two policies (which we call Explore and Control) controlling a single body and competing over the amount of observation entropy the agent experiences. The Explore agent seeks out states that maximally surprise the Control agent, which in turn aims to minimize surprise, and thereby manipulate the environment to return to familiar and predictable states. The competition between these two policies drives them to seek out increasingly surprising parts of the environment while learning to gain mastery over them. We show formally that the resulting algorithm maximizes coverage of the underlying state in block MDPs with stochastic observations, providing theoretical backing to our hypothesis that this procedure avoids uncontrollable and stochastic distractions. Our experiments further demonstrate that Adversarial Surprise leads to the emergence of complex and meaningful skills, and outperforms state-of-the-art unsupervised reinforcement learning methods in terms of both exploration and zero-shot transfer to downstream tasks.

References in corpus (20)

• Dota 2 with Large Scale Deep Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Emergent Tool Use From Multi-Agent Autocurricula
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Exploration by Random Network Distillation
• Variational Intrinsic Control
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Deep Active Inference
• Model-Based Active Exploration
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Provably efficient RL with Rich Observations via Latent State Decoding
• A unified strategy for implementing curiosity and empowerment driven reinforcement learning
• Asymmetric self-play for automatic goal discovery in robotic manipulation
• Adversarially Guided Actor-Critic
• SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Efficient Empowerment