Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces

Abstract

Meta-reinforcement learning (RL) addresses the problem of sample inefficiency in deep RL by using experience obtained in past tasks for a new task to be solved. However, most meta-RL methods require partially or fully on-policy data, i.e., they cannot reuse the data collected by past policies, which hinders the improvement of sample efficiency. To alleviate this problem, we propose a novel off-policy meta-RL method, embedding learning and evaluation of uncertainty (ELUE). An ELUE agent is characterized by the learning of a feature embedding space shared among tasks. It learns a belief model over the embedding space and a belief-conditional policy and Q-function. Then, for a new task, it collects data by the pretrained policy, and updates its belief based on the belief model. Thanks to the belief update, the performance can be improved with a small amount of data. In addition, it updates the parameters of the neural networks to adjust the pretrained relationships when there are enough data. We demonstrate that ELUE outperforms state-of-the-art meta RL methods through experiments on meta-RL benchmarks.

arXiv comment14pages

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• The information bottleneck method
• Dota 2 with Large Scale Deep Reinforcement Learning
• On First-Order Meta-Learning Algorithms
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• A Study on Overfitting in Deep Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
• Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck
• Meta reinforcement learning as task inference
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Guided Meta-Policy Search
• Model-based Adversarial Meta-Reinforcement Learning

Cited by in corpus (1)

• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments