Learning in Sparse Rewards settings through Quality-Diversity algorithms

Abstract

In the Reinforcement Learning (RL) framework, the learning is guided through a reward signal. This means that in situations of sparse rewards the agent has to focus on exploration, in order to discover which action, or set of actions leads to the reward. RL agents usually struggle with this. Exploration is the focus of Quality-Diversity (QD) methods. In this thesis, we approach the problem of sparse rewards with these algorithms, and in particular with Novelty Search (NS). This is a method that only focuses on the diversity of the possible policies behaviors. The first part of the thesis focuses on learning a representation of the space in which the diversity of the policies is evaluated. In this regard, we propose the TAXONS algorithm, a method that learns a low-dimensional representation of the search space through an AutoEncoder. While effective, TAXONS still requires information on when to capture the observation used to learn said space. For this, we study multiple ways, and in particular the signature transform, to encode information about the whole trajectory of observations. The thesis continues with the introduction of the SERENE algorithm, a method that can efficiently focus on the interesting parts of the search space. This method separates the exploration of the search space from the exploitation of the reward through a two-alternating-steps approach. The exploration is performed through NS. Any discovered reward is then locally exploited through emitters. The third and final contribution combines TAXONS and SERENE into a single approach: STAX. Throughout this thesis, we introduce methods that lower the amount of prior information needed in sparse rewards settings. These contributions are a promising step towards the development of methods that can autonomously explore and find high-performance policies in a variety of sparse rewards settings.

arXiv commentPhD Thesis

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• End-to-End Training of Deep Visuomotor Policies
• Dota 2 with Large Scale Deep Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• The CMA Evolution Strategy: A Tutorial
• OpenAI Gym
• DeepMind Control Suite
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Illuminating search spaces by mapping elites
• Large-Scale Study of Curiosity-Driven Learning
• A survey of dimensionality reduction techniques
• Visual Reinforcement Learning with Imagined Goals
• State Representation Learning for Control: An Overview
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Go-Explore: a New Approach for Hard-Exploration Problems
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space
• Reset-free Trial-and-Error Learning for Robot Damage Recovery
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• A survey on intrinsic motivation in reinforcement learning
• Sparse arrays of signatures for online character recognition
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Transforming Exploratory Creativity with DeLeNoX
• Independently Controllable Factors
• Deep Signature Transforms
• Dealing with Sparse Rewards in Reinforcement Learning
• Hierarchical Behavioral Repertoires with Unsupervised Descriptors
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition