Options as responses: Grounding behavioural hierarchies in multi-agent RL

Abstract

This paper investigates generalisation in multi-agent games, where the generality of the agent can be evaluated by playing against opponents it hasn't seen during training. We propose two new games with concealed information and complex, non-transitive reward structure (think rock/paper/scissors). It turns out that most current deep reinforcement learning methods fail to efficiently explore the strategy space, thus learning policies that generalise poorly to unseen opponents. We then propose a novel hierarchical agent architecture, where the hierarchy is grounded in the game-theoretic structure of the game -- the top level chooses strategic responses to opponents, while the low level implements them into policy over primitive actions. This grounding facilitates credit assignment across the levels of hierarchy. Our experiments show that the proposed hierarchical agent is capable of generalisation to unseen opponents, while conventional baselines fail to generalise whatsoever.

arXiv commentFirst two authors contributed equally

References in corpus (14)

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Counterfactual Multi-Agent Policy Gradients
• Emergent Tool Use From Multi-Agent Autocurricula
• Hindsight Experience Replay
• FeUdal Networks for Hierarchical Reinforcement Learning
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Hierarchical visuomotor control of humanoids
• Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research
• Open-ended Learning in Symmetric Zero-sum Games
• Re-evaluating Evaluation
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Hindsight policy gradients
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning

Cited by in corpus (4)

• RODE: Learning Roles to Decompose Multi-Agent Tasks
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Correcting Experience Replay for Multi-Agent Communication
• Hierarchical Expert Networks for Meta-Learning