Contingency-Aware Exploration in Reinforcement Learning

Abstract

This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning. To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE). In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games. The ADM is trained in a self-supervised fashion to predict the actions taken by the agent. The learned contingency information is used as a part of the state representation for exploration purposes. We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards. For example, we report a state-of-the-art score of >11,000 points on Montezuma's Revenge without using expert demonstrations, explicit high-level information (e.g., RAM states), or supervisory data. Our experiments confirm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

arXiv commentIn ICLR 2019

References in corpus (19)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Natural Scales in Geographical Patterns
• VIME: Variational Information Maximizing Exploration
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Learning to Navigate in Complex Environments
• Large-Scale Study of Curiosity-Driven Learning
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Revisiting k-means: New Algorithms via Bayesian Nonparametrics
• Unifying Count-Based Exploration and Intrinsic Motivation
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Diversity is All You Need: Learning Skills without a Reward Function
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Learning to Navigate in Cities Without a Map
• Independently Controllable Features
• Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World

Cited by in corpus (20)

• Exploration in Deep Reinforcement Learning: A Survey
• Agent57: Outperforming the Atari Human Benchmark
• First return, then explore
• Go-Explore: a New Approach for Hard-Exploration Problems
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Unsupervised State Representation Learning in Atari
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Temporal Difference Uncertainties as a Signal for Exploration
• Learning Markov State Abstractions for Deep Reinforcement Learning
• A Max-Min Entropy Framework for Reinforcement Learning
• Disentangling causal effects for hierarchical reinforcement learning
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Linear Disentangled Representations and Unsupervised Action Estimation
• Mega-Reward: Achieving Human-Level Play without Extrinsic Rewards
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Object-Oriented Dynamics Learning through Multi-Level Abstraction
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation