Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
arXiv:2001.00248
Abstract
We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph which describes a set of subtasks and their dependencies that are unknown to the agent. The agent needs to quickly adapt to the task over few episodes during adaptation phase to maximize the return in the test phase. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference(MSGI), which infers the latent parameter of the task by interacting with the environment and maximizes the return given the latent parameter. To facilitate learning, we adopt an intrinsic reward inspired by upper confidence bound (UCB) that encourages efficient exploration. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter, and to adapt more efficiently than existing meta RL and hierarchical RL methods.
arXiv commentPublished in ICLR 2020
References in corpus (15)
- Scikit-learn: Machine Learning in Python
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- StarCraft II: A New Challenge for Reinforcement Learning
- On First-Order Meta-Learning Algorithms
- Learning to reinforcement learn
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Modular Multitask Reinforcement Learning with Policy Sketches
- Harnessing Deep Neural Networks with Logic Rules
- Hybrid Reward Architecture for Reinforcement Learning
- Meta-Reinforcement Learning of Structured Exploration Strategies
- Gated-Attention Architectures for Task-Oriented Language Grounding
- Neural Logic Machines
- Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
- Autonomous Extracting a Hierarchical Structure of Tasks in Reinforcement Learning and Multi-task Reinforcement Learning