Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
arXiv:1901.10113
Abstract
Recurrent neural networks (RNNs) for reinforcement learning (RL) have shown distinct advantages, e.g., solving memory-dependent tasks and meta-learning. However, little effort has been spent on improving RNN architectures and on understanding the underlying neural mechanisms for performance gain. In this paper, we propose a novel, multiple-timescale, stochastic RNN for RL. Empirical results show that the network can autonomously learn to abstract sub-goals and can self-develop an action hierarchy using internal dynamics in a challenging continuous control task. Furthermore, we show that the self-developed compositionality of the network enhances faster re-learning when adapting to a new task that is a re-composition of previously learned sub-goals, than when starting from scratch. We also found that improved performance can be achieved when neural activities are subject to stochastic rather than deterministic dynamics.
References in corpus (16)
- Continuous control with deep reinforcement learning
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- A Recurrent Latent Variable Model for Sequential Data
- Noisy Networks for Exploration
- Recurrent World Models Facilitate Policy Evolution
- Model-Based Reinforcement Learning for Atari
- Learning to reinforcement learn
- Memory-based control with recurrent neural networks
- Bayesian Model-Agnostic Meta-Learning
- Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
- Multi-Level Discovery of Deep Options
- Learning Abstract Options
- One Big Net For Everything