Multi-task Batch Reinforcement Learning with Metric Learning

Abstract

We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards. Because the different datasets may have state-action distributions with large divergence, the task inference module can learn to ignore the rewards and spuriously correlate $\text{only}$ state-action pairs to the task identity, leading to poor test time performance. To robustify task inference, we propose a novel application of the triplet loss. To mine hard negative examples, we relabel the transitions from the training tasks by approximating their reward functions. When we allow further training on the unseen tasks, using the trained policy as an initialization leads to significantly faster convergence compared to randomly initialized policies (up to $80\%$ improvement and across 5 different Mujoco task distributions). We name our method $\text{MBML}$ ($\text{M}\text{ulti-task}$ $\text{B}\text{atch}$ RL with $\text{M}\text{etric}$ $\text{L}\text{earning}$).

First two authors contributed equally

References in corpus (40)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• In Defense of the Triplet Loss for Person Re-Identification
• An Introduction to Variational Autoencoders
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Addressing Function Approximation Error in Actor-Critic Methods
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Graph Neural Networks: A Review of Methods and Applications
• Conservative Q-Learning for Offline Reinforcement Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Visualizing and Understanding Convolutional Networks
• Learning to reinforcement learn
• Generative Adversarial Imitation Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Variational Information Bottleneck
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Fast Context Adaptation via Meta-Learning
• Causal Confusion in Imitation Learning
• CycleGAN, a Master of Steganography
• Evolved Policy Gradients
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Multi-Task Reinforcement Learning with Soft Modularization
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Meta reinforcement learning as task inference
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Distilling Policy Distillation
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Mapping State Space using Landmarks for Universal Goal Reaching
• Meta-Q-Learning
• Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
• Divide-and-Conquer Reinforcement Learning
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• Meta-Amortized Variational Inference and Learning
• ProMP: Proximal Meta-Policy Search

Cited by in corpus (5)

• Offline Meta Learning of Exploration
• Machine Learning for Robotic Manipulation
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction