Decision Transformer: Reinforcement Learning via Sequence Modeling

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

First two authors contributed equally. Last two authors advised equally

References in corpus (11)

• Language Models are Few-Shot Learners
• Zero-Shot Text-to-Image Generation
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Imitating Interactive Intelligence
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Reward-Conditioned Policies
• Synthetic Returns for Long-Term Credit Assignment
• Transformers for One-Shot Visual Imitation

Cited by in corpus (11)

• Offline Reinforcement Learning with Implicit Q-Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Shaking the foundations: delusions in sequence models for interaction and control
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• Contrastive Reinforcement Learning of Symbolic Reasoning Domains
• An Offline Deep Reinforcement Learning for Maintenance Decision-Making
• A2I Transformer: Permutation-equivariant attention network for pairwise and many-body interactions with minimal featurization
• Transfer learning with causal counterfactual reasoning in Decision Transformers