Generalized Decision Transformer for Offline Hindsight Information Matching

Abstract

How to extract as much learning signal from each trajectory data has been a key problem in reinforcement learning (RL), where sample inefficiency has posed serious challenges for practical applications. Recent works have shown that using expressive policy function approximators and conditioning on future trajectory information -- such as future states in hindsight experience replay or returns-to-go in Decision Transformer (DT) -- enables efficient learning of multi-task policies, where at times online RL is fully replaced by offline behavioral cloning, e.g. sequence modeling. We demonstrate that all these approaches are doing hindsight information matching (HIM) -- training policies that can output the rest of trajectory that matches some statistics of future state information. We present Generalized Decision Transformer (GDT) for solving any HIM problem, and show how different choices for the feature function and the anti-causal aggregator not only recover DT as a special case, but also lead to novel Categorical DT (CDT) and Bi-directional DT (BDT) for matching different statistics of the future. For evaluating CDT and BDT, we define offline multi-task state-marginal matching (SMM) and imitation learning (IL) as two generic HIM problems, propose a Wasserstein distance loss as a metric for both, and empirically study them on MuJoCo continuous control benchmarks. CDT, which simply replaces anti-causal summation with anti-causal binning in DT, enables the first effective offline multi-task SMM algorithm that generalizes well to unseen and even synthetic multi-modal state-feature distributions. BDT, which uses an anti-causal second transformer as the aggregator, can learn to model any statistics of the future and outperforms DT variants in offline multi-task IL. Our generalized formulations from HIM and GDT greatly expand the role of powerful sequence modeling architectures in modern RL.

arXiv commentAccepted to ICLR2022, Spotlight. Website: https://sites.google.com/view/generalizeddt and Code: https://github.com/frt03/generalized_dt

References in corpus (44)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Continuous control with deep reinforcement learning
• Learning Transferable Visual Models From Natural Language Supervision
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Zero-Shot Text-to-Image Generation
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Learning to reinforcement learn
• Generative Moment Matching Networks
• Learning Continuous Control Policies by Stochastic Value Gradients
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Continuous Deep Q-Learning with Model-based Acceleration
• Distributed Distributional Deterministic Policy Gradients
• Visual Reinforcement Learning with Imagined Goals
• One-Shot Visual Imitation Learning via Meta-Learning
• Behavior Regularized Offline Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Dopamine: A Research Framework for Deep Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Distributional Reinforcement Learning with Quantile Regression
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
• Imitation Learning via Off-Policy Distribution Matching
• Generalized Hindsight for Reinforcement Learning
• Training Agents using Upside-Down Reinforcement Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Multi-task Maximum Entropy Inverse Reinforcement Learning
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Offline Meta Learning of Exploration
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment
• Policy Gradients Incorporating the Future
• Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximization