Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning

Abstract

Meta-learning for offline reinforcement learning (OMRL) is an understudied problem with tremendous potential impact by enabling RL algorithms in many real-world applications. A popular solution to the problem is to infer task identity as augmented state using a context-based encoder, for which efficient learning of robust task representations remains an open challenge. In this work, we provably improve upon one of the SOTA OMRL algorithms, FOCAL, by incorporating intra-task attention mechanism and inter-task contrastive learning objectives, to robustify task representation learning against sparse reward and distribution shift. Theoretical analysis and experiments are presented to demonstrate the superior performance and robustness of our end-to-end and model-free framework compared to prior algorithms across multiple meta-RL benchmarks.

arXiv comment21 pages, 7 figures

References in corpus (15)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Recurrent Models of Visual Attention
• A Simple Neural Attentive Meta-Learner
• Learning to reinforcement learn
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Behavior Regularized Offline Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Multi-task Batch Reinforcement Learning with Metric Learning
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environments
• Adaptive Transformers in RL