Concurrent Meta Reinforcement Learning

Abstract

State-of-the-art meta reinforcement learning algorithms typically assume the setting of a single agent interacting with its environment in a sequential manner. A negative side-effect of this sequential execution paradigm is that, as the environment becomes more and more challenging, and thus requiring more interaction episodes for the meta-learner, it needs the agent to reason over longer and longer time-scales. To combat the difficulty of long time-scale credit assignment, we propose an alternative parallel framework, which we name "Concurrent Meta-Reinforcement Learning" (CMRL), that transforms the temporal credit assignment problem into a multi-agent reinforcement learning one. In this multi-agent setting, a set of parallel agents are executed in the same environment and each of these "rollout" agents are given the means to communicate with each other. The goal of the communication is to coordinate, in a collaborative manner, the most efficient exploration of the shared task the agents are currently assigned. This coordination therefore represents the meta-learning aspect of the framework, as each agent can be assigned or assign itself a particular section of the current task's state space. This framework is in contrast to standard RL methods that assume that each parallel rollout occurs independently, which can potentially waste computation if many of the rollouts end up sampling the same part of the state space. Furthermore, the parallel setting enables us to define several reward sharing functions and auxiliary losses that are non-trivial to apply in the sequential setting. We demonstrate the effectiveness of our proposed CMRL at improving over sequential methods in a variety of challenging tasks.

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Model-Free Episodic Control
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Neural Episodic Control
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Been There, Done That: Meta-Learning with Episodic Recall

Cited by in corpus (6)

• Solving Rubik's Cube with a Robot Hand
• Efficient Exploration via State Marginal Matching
• Graph Policy Gradients for Large Scale Robot Control
• Online Meta-Learning for Multi-Source and Semi-Supervised Domain Adaptation
• Represented Value Function Approach for Large Scale Multi Agent Reinforcement Learning
• PAC Guarantees for Cooperative Multi-Agent Reinforcement Learning with Restricted Communication