Learning to Communicate Implicitly By Actions

Abstract

In situations where explicit communication is limited, human collaborators act by learning to: (i) infer meaning behind their partner's actions, and (ii) convey private information about the state to their partner implicitly through actions. The first component of this learning process has been well-studied in multi-agent systems, whereas the second --- which is equally crucial for successful collaboration --- has not. To mimic both components mentioned above, thereby completing the learning process, we introduce a novel algorithm: Policy Belief Learning (PBL). PBL uses a belief module to model the other agent's private information and a policy module to form a distribution over actions informed by the belief module. Furthermore, to encourage communication by actions, we propose a novel auxiliary reward which incentivizes one agent to help its partner to make correct inferences about its private information. The auxiliary reward for communication is integrated into the learning of the policy module. We evaluate our approach on a set of environments including a matrix game, particle environment and the non-competitive bidding problem from contract bridge. We show empirically that this auxiliary reward is effective and easy to generalize. These results demonstrate that our PBL algorithm can produce strong pairs of agents in collaborative games where explicit communication is disabled.

arXiv commentAAAI 2020

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Cooperative Inverse Reinforcement Learning
• Learning Multiagent Communication with Backpropagation
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Multi-Agent Cooperation and the Emergence of (Natural) Language
• Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning
• Machine Theory of Mind
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Learning to Share and Hide Intentions using Information Regularization
• Learning Reciprocity in Complex Sequential Social Dilemmas

Cited by in corpus (6)

• Multi-Agent Common Knowledge Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Compositional ADAM: An Adaptive Compositional Solver
• A Regularized Opponent Model with Maximum Entropy Objective
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning
• BridgeHand2Vec Bridge Hand Representation