Novel Policy Seeking with Constrained Optimization
arXiv:2005.10696
Abstract
In problem-solving, we humans can come up with multiple novel solutions to the same problem. However, reinforcement learning algorithms can only produce a set of monotonous policies that maximize the cumulative reward but lack diversity and novelty. In this work, we address the problem of generating novel policies in reinforcement learning tasks. Instead of following the multi-objective framework used in existing methods, we propose to rethink the problem under a novel perspective of constrained optimization. We first introduce a new metric to evaluate the difference between policies and then design two practical novel policy generation methods following the new perspective. The two proposed methods, namely the Constrained Task Novel Bisector (CTNB) and the Interior Policy Differentiation (IPD), are derived from the feasible direction method and the interior point method commonly known in the constrained optimization literature. Experimental comparisons on the MuJoCo control suite show our methods can achieve substantial improvement over previous novelty-seeking methods in terms of both the novelty of policies and their performances in the primal task.
References in corpus (14)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Parameter Space Noise for Exploration
- Large-Scale Study of Curiosity-Driven Learning
- Deep Reinforcement Learning that Matters
- Count-Based Exploration with Neural Density Models
- Exploration by Random Network Distillation
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Stochastic Neural Networks for Hierarchical Reinforcement Learning
- Dynamics-Aware Unsupervised Discovery of Skills
- Better Exploration with Optimistic Actor-Critic
- Learning Self-Imitating Diverse Policies
- Learning Novel Policies For Tasks