Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives

Abstract

Reinforcement learning agents that operate in diverse and complex environments can benefit from the structured decomposition of their behavior. Often, this is addressed in the context of hierarchical reinforcement learning, where the aim is to decompose a policy into lower-level primitives or options, and a higher-level meta-policy that triggers the appropriate behaviors for a given situation. However, the meta-policy must still produce appropriate decisions in all states. In this work, we propose a policy design that decomposes into primitives, similarly to hierarchical reinforcement learning, but without a high-level meta-policy. Instead, each primitive can decide for themselves whether they wish to act in the current state. We use an information-theoretic mechanism for enabling this decentralized decision: each primitive chooses how much information it needs about the current state to make a decision and the primitive that requests the most information about the current state acts in the world. The primitives are regularized to use as little information as possible, which leads to natural competition and specialization. We experimentally demonstrate that this policy architecture improves over both flat and hierarchical policies in terms of generalization.

arXiv commentPreprint, Under Review

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Emergence of Locomotion Behaviours in Rich Environments
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Deep Variational Information Bottleneck
• Modular Multitask Reinforcement Learning with Policy Sketches
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Meta Learning Shared Hierarchies
• Neural probabilistic motor primitives for humanoid control
• Latent Space Policies for Hierarchical Reinforcement Learning
• Routing Networks and the Challenges of Modular and Compositional Computation

Cited by in corpus (7)

• Recurrent Independent Mechanisms
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• SOAC: The Soft Option Actor-Critic Architecture
• Decentralized Reinforcement Learning: Global Decision-Making via Local Economic Transactions
• Disentangling causal effects for hierarchical reinforcement learning
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning