Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems

Abstract

We report a novel, computationally efficient approach for solving hard nonlinear problems of reinforcement learning (RL). Here we combine umbrella sampling, from computational physics/chemistry, with optimal control methods. The approach is realized on the basis of neural networks, with the use of policy gradient. It outperforms, by computational efficiency and implementation universality, all available state-of-the-art algorithms, in application to hard RL problems with sparse reward, state traps and lack of terminal states. The proposed approach uses an ensemble of simultaneously acting agents, with a modified reward which includes the ensemble entropy, yielding an optimal exploration-exploitation balance.

References in corpus (5)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Benchmarking Model-Based Reinforcement Learning
• First return, then explore
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning

Cited by in corpus (1)

• An effective control of large systems of active particles: An application to evacuation problem