Enhanced Scene Specificity with Sparse Dynamic Value Estimation

Abstract

Multi-scene reinforcement learning involves training the RL agent across multiple scenes / levels from the same task, and has become essential for many generalization applications. However, the inclusion of multiple scenes leads to an increase in sample variance for policy gradient computations, often resulting in suboptimal performance with the direct application of traditional methods (e.g. PPO, A3C). One strategy for variance reduction is to consider each scene as a distinct Markov decision process (MDP) and learn a joint value function dependent on both state (s) and MDP (M). However, this is non-trivial as the agent is usually unaware of the underlying level at train / test times in multi-scene RL. Recently, Singh et al. [1] tried to address this by proposing a dynamic value estimation approach that models the true joint value function distribution as a Gaussian mixture model (GMM). In this paper, we argue that the error between the true scene-specific value function and the predicted dynamic estimate can be further reduced by progressively enforcing sparse cluster assignments once the agent has explored most of the state space. The resulting agents not only show significant improvements in the final reward score across a range of OpenAI ProcGen environments, but also exhibit increased navigation efficiency while completing a game level.

References in corpus (5)

• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
• An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning