Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning

Abstract

Meta-reinforcement learning typically requires orders of magnitude more samples than single task reinforcement learning methods. This is because meta-training needs to deal with more diverse distributions and train extra components such as context encoders. To address this, we propose a novel self-supervised learning task, which we named Trajectory Contrastive Learning (TCL), to improve meta-training. TCL adopts contrastive learning and trains a context encoder to predict whether two transition windows are sampled from the same trajectory. TCL leverages the natural hierarchical structure of context-based meta-RL and makes minimal assumptions, allowing it to be generally applicable to context-based meta-RL algorithms. It accelerates the training of context encoders and improves meta-training overall. Experiments show that TCL performs better or comparably than a strong meta-RL baseline in most of the environments on both meta-RL MuJoCo (5 of 6) and Meta-World benchmarks (44 out of 50).

References in corpus (21)

• A Simple Framework for Contrastive Learning of Visual Representations
• Prototypical Networks for Few-shot Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Theoretical Models of Learning to Learn
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Exploring Simple Siamese Representation Learning
• Learning to reinforcement learn
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Towards a Neural Statistician
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
• Learning to Explore with Meta-Policy Gradient
• Learning Actionable Representations from Visual Observations
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• ProMP: Proximal Meta-Policy Search