Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
arXiv:2102.07266
Abstract
Training deep reinforcement learning agents on environments with multiple levels / scenes from the same task, has become essential for many applications aiming to achieve generalization and domain transfer from simulation to the real world. While such a strategy is helpful with generalization, the use of multiple scenes significantly increases the variance of samples collected for policy gradient computations. Current methods, effectively continue to view this collection of scenes as a single Markov decision process (MDP), and thus learn a scene-generic value function V(s). However, we argue that the sample variance for a multi-scene environment is best minimized by treating each scene as a distinct MDP, and then learning a joint value function V(s,M) dependent on both state s and MDP M. We further demonstrate that the true joint value function for a multi-scene environment, follows a multi-modal distribution which is not captured by traditional CNN / LSTM based critic networks. To this end, we propose a dynamic value estimation (DVE) technique, which approximates the true joint value function through a sparse attention mechanism over multiple value function hypothesis / modes. The resulting agent not only shows significant improvements in the final reward score across a range of OpenAI ProcGen environments, but also exhibits enhanced navigation efficiency and provides an implicit mechanism for unsupervised state-space skill decomposition.
arXiv commentThis work is a merger of arXiv:2005.12254 and arXiv:2011.12574
References in corpus (19)
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- On Evaluation of Embodied Navigation Agents
- Deep Exploration via Bootstrapped DQN
- Learning to reinforcement learn
- AI2-THOR: An Interactive 3D Environment for Visual AI
- A Study on Overfitting in Deep Reinforcement Learning
- Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Distributional Reinforcement Learning with Quantile Regression
- EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
- Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Recurrent Independent Mechanisms
- Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
- An Empirical Study on Hyperparameters and their Interdependence for RL Generalization
- VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control