Inter-Level Cooperation in Hierarchical Reinforcement Learning
arXiv:1912.02368
Abstract
Hierarchies of temporally decoupled policies present a promising approach for enabling structured exploration in complex long-term planning problems. To fully achieve this approach an end-to-end training paradigm is needed. However, training these multi-level policies has had limited success due to challenges arising from interactions between the goal-assigning and goal-achieving levels within a hierarchy. In this article, we consider the policy optimization process as a multi-agent process. This allows us to draw on connections between communication and cooperation in multi-agent RL, and demonstrate the benefits of increased cooperation between sub-policies on the training performance of the overall policy. We introduce a simple yet effective technique for inducing inter-level cooperation by modifying the objective function and subsequent gradients of higher-level policies. Experimental results on a wide variety of simulated robotics and traffic control tasks demonstrate that inducing cooperation results in stronger performing policies and increased sample efficiency on a set of difficult long time horizon tasks. We also find that goal-conditioned policies trained using our method display better transfer to new tasks, highlighting the benefits of our method in learning task-agnostic lower-level behaviors. Videos and code are available at: https://sites.google.com/berkeley.edu/cooperative-hrl.
References in corpus (18)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- A Survey and Critique of Multiagent Deep Reinforcement Learning
- Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
- Learning Multiagent Communication with Backpropagation
- Data-Efficient Hierarchical Reinforcement Learning
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- Stochastic Neural Networks for Hierarchical Reinforcement Learning
- Constrained Policy Optimization
- Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
- Biases for Emergent Communication in Multi-agent Reinforcement Learning
- Sub-policy Adaptation for Hierarchical Reinforcement Learning
- Terrain RL Simulator