Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Abstract

We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.

References in corpus (6)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

Cited by in corpus (251)

• A Brief Survey of Deep Reinforcement Learning
• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver
• PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Mean Field Multi-Agent Reinforcement Learning
• Learning Attentional Communication for Multi-Agent Cooperation
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Cooperative Multi-Agent Deep Reinforcement Learning for Reliable Surveillance via Autonomous Multi-UAV Control
• The Hanabi Challenge: A New Frontier for AI Research
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• A Deep Reinforcement Learning Chatbot
• Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence
• Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
• Multi-Objective Deep Reinforcement Learning
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Federated Deep Reinforcement Learning
• Reward Design for Driver Repositioning Using Multi-Agent Reinforcement Learning
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Risk-Aware Energy Scheduling for Edge Computing with Microgrid: A Multi-Agent Deep Reinforcement Learning Approach
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Multiagent Soft Q-Learning
• Compositional Obverter Communication Learning From Raw Visual Input
• Multi-Agent Common Knowledge Reinforcement Learning
• Improving Coordination in Small-Scale Multi-Agent Deep Reinforcement Learning through Memory-driven Communication
• A Deep Reinforcement Learning Framework for Contention-Based Spectrum Sharing
• Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Deep Coordination Graphs
• Towards Joint Learning of Optimal MAC Signaling and Wireless Channel Access
• PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning
• Robust Multi-agent Communication via Multi-view Message Certification
• Online Multi-agent Reinforcement Learning for Decentralized Inverter-based Volt-VAR Control
• Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control
• Semantics-Native Communication with Contextual Reasoning
• Learning Efficient Multi-agent Communication: An Information Bottleneck Approach
• Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
• Factorized Q-Learning for Large-Scale Multi-Agent Systems
• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Emergence of Compositional Language with Deep Generational Transmission
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning
• Emergent Multi-Agent Communication in the Deep Learning Era
• "Other-Play" for Zero-Shot Coordination
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• On the interaction between supervision and self-play in emergent communication
• Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Games
• A non-cooperative meta-modeling game for automated third-party calibrating, validating, and falsifying constitutive laws with parallelized adversarial attacks
• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• Goal-Oriented Scheduling in Sensor Networks with Application Timing Awareness
• AI Research Considerations for Human Existential Safety (ARCHES)
• Developmentally motivated emergence of compositional communication via template transfer
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Fair collaborative vehicle routing: A deep multi-agent reinforcement learning approach
• Cooperative Multi-Agent Reinforcement Learning for Low-Level Wireless Communication
• Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning
• Explaining Deep Neural Networks
• A Generalized Training Approach for Multiagent Learning
• Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning
• The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning
• Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• Multi-Agent Collaboration via Reward Attribution Decomposition
• Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances
• Demand-Side Scheduling Based on Multi-Agent Deep Actor-Critic Learning for Smart Grids
• A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning
• Collaborative Visual Navigation
• Learning to Collocate Neural Modules for Image Captioning
• Message Passing Multi-Agent GANs
• Learning Distilled Collaboration Graph for Multi-Agent Perception
• Ease-of-Teaching and Language Structure from Emergent Communication
• Deep Reinforcement One-Shot Learning for Artificially Intelligent Classification Systems
• Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learning
• Concurrent Meta Reinforcement Learning
• DarKnight: A Data Privacy Scheme for Training and Inference of Deep Neural Networks
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent Reinforcement Learning
• Learning Multi-Agent Coordination through Connectivity-driven Communication
• Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning
• Policy Gradient With Value Function Approximation For Collective Multiagent Planning
• Multi-agent Hierarchical Reinforcement Learning with Dynamic Termination
• Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning
• Toward Packet Routing with Fully-distributed Multi-agent Deep Reinforcement Learning
• Message-Aware Graph Attention Networks for Large-Scale Multi-Robot Path Planning
• Learning Latent Representations to Influence Multi-Agent Interaction
• Deep Reinforcement Learning for Green Security Games with Real-Time Information
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• Learning Latent Representations to Co-Adapt to Humans
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Building Human-like Communicative Intelligence: A Grounded Perspective
• Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning
• Emergence of Pragmatics from Referential Game between Theory of Mind Agents
• Fairness-aware Competitive Bidding Influence Maximization in Social Networks
• Reinforcement Learning Under Moral Uncertainty
• End-to-End Simultaneous Speech Translation with Differentiable Segmentation
• STMARL: A Spatio-Temporal Multi-Agent Reinforcement Learning Approach for Cooperative Traffic Light Control
• M${}^3$RL: Mind-aware Multi-agent Management Reinforcement Learning
• Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Interpretable and Pedagogical Examples
• Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning
• Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity
• Networked Multi-Agent Reinforcement Learning with Emergent Communication
• Emergence of Theory of Mind Collaboration in Multiagent Systems
• The Emergence of Wireless MAC Protocols with Multi-Agent Reinforcement Learning
• A Multi-Agent Reinforcement Learning Method for Impression Allocation in Online Display Advertising
• Emergent Language: A Survey and Taxonomy
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• Learning from My Partner's Actions: Roles in Decentralized Robot Teams
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Computational Performance of Deep Reinforcement Learning to find Nash Equilibria
• Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
• Learning Multi-Agent Intention-Aware Communication for Optimal Multi-Order Execution in Finance
• Separation of Concerns in Reinforcement Learning
• Emergent Road Rules In Multi-Agent Driving Environments
• Joint Policy Search for Multi-agent Collaboration with Imperfect Information
• Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation
• Learning to Communicate in Multi-Agent Reinforcement Learning : A Review
• Learning Autocomplete Systems as a Communication Game
• Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning
• Learning to Communicate Implicitly By Actions
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• Grounding Artificial Intelligence in the Origins of Human Behavior
• Collective Intelligence for Deep Learning: A Survey of Recent Developments
• Calibration of Shared Equilibria in General Sum Partially Observable Markov Games
• Incorporating Pragmatic Reasoning Communication into Emergent Language
• A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learning
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications
• Inter-Cell Network Slicing With Transfer Learning Empowered Multi-Agent Deep Reinforcement Learning
• Quasi-Equivalence Discovery for Zero-Shot Emergent Communication
• Adversarially Guided Self-Play for Adopting Social Conventions
• Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint
• Learning Agile Locomotion via Adversarial Training
• Emergent Graphical Conventions in a Visual Communication Game
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Value-Decomposition Multi-Agent Actor-Critics
• Learning to Communicate Using Counterfactual Reasoning
• A Visual Communication Map for Multi-Agent Deep Reinforcement Learning
• Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
• Re-conceptualising the Language Game Paradigm in the Framework of Multi-Agent Reinforcement Learning
• Intelligent Coordination among Multiple Traffic Intersections Using Multi-Agent Reinforcement Learning
• Stigmergic Independent Reinforcement Learning for Multi-Agent Collaboration
• Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
• Non-cooperative Multi-agent Systems with Exploring Agents
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Learning to Deceive in Multi-Agent Hidden Role Games
• Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Games
• Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
• Dynamic population-based meta-learning for multi-agent communication with natural language
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Regret Bounds for Decentralized Learning in Cooperative Multi-Agent Dynamical Systems
• Rat big, cat eaten! Ideas for a useful deep-agent protolanguage
• Finite-Sample Analysis For Decentralized Batch Multi-Agent Reinforcement Learning With Networked Agents
• Distributed Resource Allocation with Multi-Agent Deep Reinforcement Learning for 5G-V2V Communication
• What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data
• Multi-Agent Coordination in Adversarial Environments through Signal Mediated Strategies
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• Correcting Experience Replay for Multi-Agent Communication
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Collaborative Multi-Agent Video Fast-Forwarding
• Integrating LEO Satellites and Multi-UAV Reinforcement Learning for Hybrid FSO/RF Non-Terrestrial Networks
• Analogs of Linguistic Structure in Deep Representations
• Interactive Learning from Activity Description
• Blind interactive learning of modulation schemes: Multi-agent cooperation without co-design
• DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Future
• Competitive Multi-Agent Deep Reinforcement Learning with Counterfactual Thinking
• Centralized Model and Exploration Policy for Multi-Agent RL
• Communications that Emerge through Reinforcement Learning Using a (Recurrent) Neural Network
• An Overview of Natural Language State Representation for Reinforcement Learning
• Attentional Policies for Cross-Context Multi-Agent Reinforcement Learning
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Vehicle Communication Strategies for Simulated Highway Driving
• Learning to Communicate with Reinforcement Learning for an Adaptive Traffic Control System
• Towards Graph Representation Learning in Emergent Communication
• Multi-vehicle Flocking Control with Deep Deterministic Policy Gradient Method
• Distributed Policy Iteration for Scalable Approximation of Cooperative Multi-Agent Policies
• PAC Guarantees for Cooperative Multi-Agent Reinforcement Learning with Restricted Communication
• Multi-agent Cooperative Games Using Belief Map Assisted Training
• Learning Complex Multi-Agent Policies in Presence of an Adversary
• A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning
• Pommerman: A Multi-Agent Playground
• Winning an Election: On Emergent Strategic Communication in Multi-Agent Networks
• Who2com: Collaborative Perception via Learnable Handshake Communication
• Human and Multi-Agent collaboration in a human-MARL teaming framework
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Iterated learning for emergent systematicity in VQA
• Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations
• HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism
• Intelligent Link Adaptation for Grant-Free Access Cellular Networks: A Distributed Deep Reinforcement Learning Approach
• Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges
• Collaborative Deep Reinforcement Learning for Joint Object Search
• Sparse Communication via Mixed Distributions
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning
• Decentralized Multi-Agents by Imitation of a Centralized Controller
• Multi-lingual agents through multi-headed neural networks
• Improving the Generalization of Unseen Crowd Behaviors for Reinforcement Learning based Local Motion Planners
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• AB-Mapper: Attention and BicNet Based Multi-agent Path Finding for Dynamic Crowded Environment
• APES: a Python toolbox for simulating reinforcement learning environments
• The emergence of visual semantics through communication games
• Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning
• A Signal-Centric Perspective on the Evolution of Symbolic Communication
• Learning to Communicate with Strangers via Channel Randomisation Methods
• Multi-agent Collaboration for Feasible Collaborative Behavior Construction and Evaluation
• Neural networks can understand compositional functions that humans do not, in the context of emergent communication
• FireCommander: An Interactive, Probabilistic Multi-agent Environment for Heterogeneous Robot Teams
• Characterizing the Decidability of Finite State Automata Team Games with Communication
• Planning, Inference and Pragmatics in Sequential Language Games
• Generalizing Emergent Communication
• Seq2Seq Mimic Games: A Signaling Perspective
• Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi
• State Distribution-aware Sampling for Deep Q-learning
• DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
• Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning via Best Response
• Collaboration Promotes Group Resilience in Multi-Agent RL
• Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control
• Reinforced Natural Language Interfaces via Entropy Decomposition
• Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT
• BGC: Multi-Agent Group Belief with Graph Clustering
• Visual Perspective Taking for Opponent Behavior Modeling
• The Confluence of Networks, Games and Learning
• Emergent Communication with World Models
• Individual and Collective Autonomous Development
• Learning Cooperation and Online Planning Through Simulation and Graph Convolutional Network
• RACE: Reinforced Cooperative Autonomous Vehicle Collision AvoidancE
• On Memory Mechanism in Multi-Agent Reinforcement Learning
• Modeling Sensorimotor Coordination as Multi-Agent Reinforcement Learning with Differentiable Communication
• Targeted Data Acquisition for Evolving Negotiation Agents