Parallel Exploration via Negatively Correlated Search

Abstract

Effective exploration is a key to successful search. The recently proposed Negatively Correlated Search (NCS) tries to achieve this by parallel exploration, where a set of search processes are driven to be negatively correlated so that different promising areas of the search space can be visited simultaneously. Various applications have verified the advantages of such novel search behaviors. Nevertheless, the mathematical understandings are still lacking as the previous NCS was mostly devised by intuition. In this paper, a more principled NCS is presented, explaining that the parallel exploration is equivalent to the explicit maximization of both the population diversity and the population solution qualities, and can be optimally obtained by partially gradient descending both models with respect to each search process. For empirical assessments, the reinforcement learning tasks that largely demand exploration ability is considered. The new NCS is applied to the popular reinforcement learning problems, i.e., playing Atari games, to directly train a deep convolution network with 1.7 million connection weights in the environments with uncertain and delayed rewards. Empirical results show that the significant advantages of NCS over the compared state-of-the-art methods can be highly owed to the effective parallel exploration ability.

References in corpus (10)

• A Brief Survey of Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• Recurrent World Models Facilitate Policy Evolution
• Parameter Space Noise for Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• High-dimensional Black-box Optimization via Divide and Approximate Conquer
• Negatively Correlated Search
• Value Prediction Network
• Derivative-Free Reinforcement Learning: A Review
• A Parallel Divide-and-Conquer based Evolutionary Algorithm for Large-scale Optimization

Cited by in corpus (5)

• Reducing Idleness in Financial Cloud Services via Multi-objective Evolutionary Reinforcement Learning based Load Balancer
• Mitigating Unfairness via Evolutionary Multi-objective Ensemble Learning
• A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning
• Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
• QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE