Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity

Abstract

Quality-Diversity (QD) is a concept from Neuroevolution with some intriguing applications to Reinforcement Learning. It facilitates learning a population of agents where each member is optimized to simultaneously accumulate high task-returns and exhibit behavioral diversity compared to other members. In this paper, we build on a recent kernel-based method for training a QD policy ensemble with Stein variational gradient descent. With kernels based on $f$-divergence between the stationary distributions of policies, we convert the problem to that of efficient estimation of the ratio of these stationary distributions. We then study various distribution ratio estimators used previously for off-policy evaluation and imitation and re-purpose them to compute the gradients for policies in an ensemble such that the resultant population is diverse and of high-quality.

arXiv commentCoRL 2020 camera-ready

References in corpus (12)

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robots that can adapt like animals
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Illuminating search spaces by mapping elites
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
• Off-Policy Policy Gradient with State Distribution Correction
• Hardware Conditioned Policies for Multi-Robot Transfer Learning
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Learning Self-Imitating Diverse Policies
• Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning