Stabilizing Transformers for Reinforcement Learning

Abstract

Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation. Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting. In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture. We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical. GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially observable environments.

References in corpus (13)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Conditional Image Generation with PixelCNN Decoders
• Grid Long Short-Term Memory
• Model-Free Episodic Control
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Neural Episodic Control
• Meta reinforcement learning as task inference
• Relative Entropy Regularized Policy Iteration
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Multi-task Deep Reinforcement Learning with PopArt
• TF-Replicator: Distributed Machine Learning for Researchers
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning

Cited by in corpus (21)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• Grounded Language Learning Fast and Slow
• Addressing Some Limitations of Transformers with Feedback Memory
• DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation
• Representation learning for neural population activity with Neural Data Transformers
• Self-Supervised Reinforcement Learning for Recommender Systems
• Flatland-RL : Multi-Agent Reinforcement Learning on Trains
• Normalized Attention Without Probability Cage
• Transformer Based Reinforcement Learning For Games
• SUMBT+LaRL: Effective Multi-domain End-to-end Neural Task-oriented Dialog System
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Sub-Linear Memory: How to Make Performers SLiM
• Few-shot Sequence Learning with Transformers
• Adaptive Transformers in RL
• Designing a Prospective COVID-19 Therapeutic with Reinforcement Learning
• Perception-Prediction-Reaction Agents for Deep Reinforcement Learning
• Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning
• Stabilizing Transformer-Based Action Sequence Generation For Q-Learning
• Function Approximation for Reinforcement Learning Controller for Energy from Spread Waves
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation