Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning

Abstract

Context, the embedding of previous collected trajectories, is a powerful construct for Meta-Reinforcement Learning (Meta-RL) algorithms. By conditioning on an effective context, Meta-RL policies can easily generalize to new tasks within a few adaptation steps. We argue that improving the quality of context involves answering two questions: 1. How to train a compact and sufficient encoder that can embed the task-specific information contained in prior trajectories? 2. How to collect informative trajectories of which the corresponding context reflects the specification of tasks? To this end, we propose a novel Meta-RL framework called CCM (Contrastive learning augmented Context-based Meta-RL). We first focus on the contrastive nature behind different tasks and leverage it to train a compact and sufficient context encoder. Further, we train a separate exploration policy and theoretically derive a new information-gain-based objective which aims to collect informative trajectories in a few steps. Empirically, we evaluate our approaches on common benchmarks as well as several complex sparse-reward environments. The experimental results show that CCM outperforms state-of-the-art algorithms by addressing previously mentioned problems respectively.

Accepted to AAAI 2021

References in corpus (14)

• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Theoretical Models of Learning to Learn
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning to reinforcement learn
• On Variational Bounds of Mutual Information
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Environment Probing Interaction Policies
• Meta-Q-Learning
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning

Cited by in corpus (3)

• Contrastive Reinforcement Learning of Symbolic Reasoning Domains
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning