Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Abstract

The problem of sparse rewards is one of the hardest challenges in contemporary reinforcement learning. Hierarchical reinforcement learning (HRL) tackles this problem by using a set of temporally-extended actions, or options, each of which has its own subgoal. These subgoals are normally handcrafted for specific tasks. Here, though, we introduce a generic class of subgoals with broad applicability in the visual domain. Underlying our approach (in common with work using "auxiliary tasks") is the hypothesis that the ability to control aspects of the environment is an inherently useful skill to have. We incorporate such subgoals in an end-to-end hierarchical reinforcement learning system and test two variants of our algorithm on a number of games from the Atari suite. We highlight the advantage of our approach in one of the hardest games -- Montezuma's revenge -- for which the ability to handle sparse rewards is key. Our agent learns several times faster than the current state-of-the-art HRL agent in this game, reaching a similar level of performance. UPDATE 22/11/17: We found that a standard A3C agent with a simple shaped reward, i.e. extrinsic reward + feature control intrinsic reward, has comparable performance to our agent in Montezuma Revenge. In light of the new experiments performed, the advantage of our HRL approach can be attributed more to its ability to learn useful features from intrinsic rewards rather than its ability to explore and reuse abstracted skills with hierarchical components. This has led us to a new conclusion about the result.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• OpenAI Gym
• VIME: Variational Information Maximizing Exploration
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Learning to reinforcement learn
• Learning to Navigate in Complex Environments
• Count-Based Exploration with Neural Density Models
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Variational Intrinsic Control
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Independently Controllable Features

Cited by in corpus (17)

• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• A survey on intrinsic motivation in reinforcement learning
• MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Contingency-Aware Exploration in Reinforcement Learning
• RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Hierarchical Reinforcement Learning for Relay Selection and Power Optimization in Two-Hop Cooperative Relay Network
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Mutual Information State Intrinsic Control
• Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness
• Robotic self-representation improves manipulation skills and transfer learning