Decision Transformer: Reinforcement Learning via Sequence Modeling

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

arXiv commentFirst two authors contributed equally. Last two authors advised equally

References in corpus (27)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Playing Atari with Deep Reinforcement Learning
• Language Models are Few-Shot Learners
• Zero-Shot Text-to-Image Generation
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Fine-Tuning Language Models from Human Preferences
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• Reinforcement Learning from Imperfect Demonstrations
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Imitating Interactive Intelligence
• Learning to Reach Goals via Iterated Supervised Learning
• Training Agents using Upside-Down Reinforcement Learning
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Reward-Conditioned Policies
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Synthetic Returns for Long-Term Credit Assignment
• Reset-Free Lifelong Learning with Skill-Space Planning
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning
• Transformers for One-Shot Visual Imitation

Cited by in corpus (51)

• On the Opportunities and Risks of Foundation Models
• Transformers in Time-series Analysis: A Tutorial
• A Minimalist Approach to Offline Reinforcement Learning
• NetLLM: Adapting Large Language Models for Networking
• Regression Transformer: Concurrent sequence regression and generation for molecular language modeling
• Offline Reinforcement Learning with Implicit Q-Learning
• Pix2seq: A Language Modeling Framework for Object Detection
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• A Survey on Influence Maximization: From an ML-Based Combinatorial Optimization
• Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model
• Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap
• User Retention-oriented Recommendation with Decision Transformer
• Extracting Sentence Embeddings from Pretrained Transformer Models
• Generative Personas That Behave and Experience Like Humans
• On Grounded Planning for Embodied Tasks with Language Models
• Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions
• NeurDB: An AI-powered Autonomous Data System
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planning
• Secure short-term load forecasting for smart grids with transformer-based federated learning
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Optimizing Credit Limit Adjustments Under Adversarial Goals Using Reinforcement Learning
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• An empathic GPT-based chatbot to talk about mental disorders with Spanish teenagers
• Self-Supervised Learning to Prove Equivalence Between Straight-Line Programs via Rewrite Rules
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Shaking the foundations: delusions in sequence models for interaction and control
• SigFormer: Signature Transformers for Deep Hedging
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Saturn Platform: Foundation Model Operations and Generative AI for Financial Services
• What Would Jiminy Cricket Do? Towards Agents That Behave Morally
• Contrastive Reinforcement Learning of Symbolic Reasoning Domains
• Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise
• Experimental Standards for Deep Learning in Natural Language Processing Research
• An Offline Deep Reinforcement Learning for Maintenance Decision-Making
• Minimal Neural Network Models for Permutation Invariant Agents
• Offline RL With Resource Constrained Online Deployment
• Playing Non-Embedded Card-Based Games with Reinforcement Learning
• Transfer learning with causal counterfactual reasoning in Decision Transformers
• Quantile Filtered Imitation Learning
• Teaching Autoregressive Language Models Complex Tasks By Demonstration
• A2I Transformer: Permutation-equivariant attention network for pairwise and many-body interactions with minimal featurization
• Remember what you did so you know what to do next
• CubeTR: Learning to Solve The Rubiks Cube Using Transformers
• Non-maximizing policies that fulfill multi-criterion aspirations in expectation
• Medical Vision Language Models as Policies for Robotic Surgery