Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi

Abstract

The card game Hanabi is considered a strong medium for the testing and development of multi-agent reinforcement learning (MARL) algorithms, due to its cooperative nature, partial observability, limited communication and remarkable complexity. Previous research efforts have explored the capabilities of MARL algorithms within Hanabi, focusing largely on advanced architecture design and algorithmic manipulations to achieve state-of-the-art performance for various number of cooperators. However, this often leads to complex solution strategies with high computational cost and requiring large amounts of training data. For humans to solve the Hanabi game effectively, they require the use of conventions, which often allows for a means to implicitly convey ideas or knowledge based on a predefined, and mutually agreed upon, set of "rules" or principles. Multi-agent problems containing partial observability, especially when limited communication is present, can benefit greatly from the use of implicit knowledge sharing. In this paper, we propose a novel approach to augmenting an agent's action space using conventions, which act as a sequence of special cooperative actions that span over and include multiple time steps and multiple agents, requiring agents to actively opt in for it to reach fruition. These conventions are based on existing human conventions, and result in a significant improvement on the performance of existing techniques for self-play and cross-play for various number of cooperators within Hanabi.

arXiv commentThis paper is accepted and published in the journal of autonomous agents and multi-agent systems (JAAMAS). The updated manuscript is the revised version after the final round of peer revision

References in corpus (17)

• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Deep Recurrent Q-Learning for Partially Observable MDPs
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• The Hanabi Challenge: A New Frontier for AI Research
• Guided Deep Reinforcement Learning for Swarm Systems
• On the Utility of Learning about Humans for Human-AI Coordination
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• On Multi-Agent Learning in Team Sports Games
• Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs