Effective Diversity in Population Based Reinforcement Learning

Abstract

Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.

arXiv commentCamera-ready version, NeurIPS 2020

References in corpus (27)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Analysis of Thompson Sampling for the multi-armed bandit problem
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Noisy Networks for Exploration
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Illuminating search spaces by mapping elites
• Structure Discovery in Nonparametric Regression through Compositional Kernel Search
• Model-Ensemble Trust-Region Policy Optimization
• Ray: A Distributed Framework for Emerging AI Applications
• Agent57: Outperforming the Atari Human Benchmark
• Simple random search provides a competitive approach to reinforcement learning
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Emergent Coordination Through Competition
• Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
• Evolution-Guided Policy Gradient in Reinforcement Learning
• Better Exploration with Optimistic Actor-Critic
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning
• Adapting Behaviour for Learning Progress
• Ready Policy One: World Building Through Active Learning
• Population-Guided Parallel Policy Search for Reinforcement Learning
• A Generalized Framework for Population Based Training

Cited by in corpus (10)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Differentiable Quality Diversity
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• An Instance-Dependent Analysis for the Cooperative Multi-Player Multi-Armed Bandit
• Effects of Different Optimization Formulations in Evolutionary Reinforcement Learning on Diverse Behavior Generation
• Safety Aware Reinforcement Learning (SARL)
• Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
• Understanding the Effect of Stochasticity in Policy Optimization