Learning latent state representation for speeding up exploration

Abstract

Exploration is an extremely challenging problem in reinforcement learning, especially in high dimensional state and action spaces and when only sparse rewards are available. Effective representations can indicate which components of the state are task relevant and thus reduce the dimensionality of the space to explore. In this work, we take a representation learning viewpoint on exploration, utilizing prior experience to learn effective latent representations, which can subsequently indicate which regions to explore. Prior experience on separate but related tasks help learn representations of the state which are effective at predicting instantaneous rewards. These learned representations can then be used with an entropy-based exploration method to effectively perform exploration in high dimensional spaces by effectively lowering the dimensionality of the search space. We show the benefits of this representation for meta-exploration in a simulated object pushing environment.

7 pages, 8 figures, workshop

References in corpus (14)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Trust Region Policy Optimization
• Tutorial on Variational Autoencoders
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Parameter Space Noise for Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Deep Q-learning from Demonstrations
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Provably Efficient Maximum Entropy Exploration
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously

Cited by in corpus (8)

• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• A survey on intrinsic motivation in reinforcement learning
• A hybrid learning method for system identification and optimal control
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Reinforcement Learning by Guided Safe Exploration
• Reinforcement Learning with Efficient Active Feature Acquisition
• Prioritized Guidance for Efficient Multi-Agent Reinforcement Learning Exploration
• ExTra: Transfer-guided Exploration