Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

Abstract

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL.

Camera-ready version, International Conference of Machine Learning 2017; updated to fix print-breaking image

References in corpus (5)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Opponent Modeling in Deep Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games

Cited by in corpus (94)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• On Tiny Episodic Memories in Continual Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• A Survey of Deep Reinforcement Learning in Video Games
• Game-Theoretic Multiagent Reinforcement Learning
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System
• UAV Swarm Path Planning with Reinforcement Learning for Field prospecting
• Multiagent Soft Q-Learning
• The Multi-Agent Reinforcement Learning in MalmÖ (MARLÖ) Competition
• Multi-Agent Common Knowledge Reinforcement Learning
• The NetHack Learning Environment
• A Deep Policy Inference Q-Network for Multi-Agent Systems
• Deep Coordination Graphs
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Towards Joint Learning of Optimal MAC Signaling and Wireless Channel Access
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction
• Reconfigurable Intelligent Surface Assisted Device-to-Device Communications
• "Other-Play" for Zero-Shot Coordination
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Spatial Intention Maps for Multi-Agent Mobile Manipulation
• TorchBeast: A PyTorch Platform for Distributed RL
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Network slicing for vehicular communications: a multi-agent deep reinforcement learning approach
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Value Propagation Networks
• Towards Cooperation in Sequential Prisoner's Dilemmas: a Deep Multiagent Reinforcement Learning Approach
• Enhancing the Performance of Multi-Agent Reinforcement Learning for Controlling HVAC Systems
• Toward Packet Routing with Fully-distributed Multi-agent Deep Reinforcement Learning
• Online Class-Incremental Continual Learning with Adversarial Shapley Value
• Multi-agent Hierarchical Reinforcement Learning with Dynamic Termination
• Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Multi-Agent Reinforcement Learning via Distributed MPC as a Function Approximator
• Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning
• Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-level
• PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning
• Deep Learning based Wireless Resource Allocation with Application to Vehicular Networks
• Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning
• Emergence of Theory of Mind Collaboration in Multiagent Systems
• A Sufficient Statistic for Influence in Structured Multiagent Environments
• Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Influencing Towards Stable Multi-Agent Interactions
• Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios
• Calibration of Shared Equilibria in General Sum Partially Observable Markov Games
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• Efficient Ridesharing Dispatch Using Multi-Agent Reinforcement Learning
• A game-theoretic analysis of networked system control for common-pool resource management using multi-agent reinforcement learning
• Value-Decomposition Multi-Agent Actor-Critics
• MARL with General Utilities via Decentralized Shadow Reward Actor-Critic
• Online Multi-Objective Model-Independent Adaptive Tracking Mechanism for Dynamical Systems
• Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning
• Heterogeneous Graph Attention Networks for Learning Diverse Communication
• Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning
• Inter-Level Cooperation in Hierarchical Reinforcement Learning
• Correcting Experience Replay for Multi-Agent Communication
• Integrating LEO Satellites and Multi-UAV Reinforcement Learning for Hybrid FSO/RF Non-Terrestrial Networks
• Offline Decentralized Multi-Agent Reinforcement Learning
• MAMRL: Exploiting Multi-agent Meta Reinforcement Learning in WAN Traffic Engineering
• Intent-aware Multi-agent Reinforcement Learning
• Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
• Vehicular Network Slicing for Reliable Access and Deadline-Constrained Data Offloading: A Multi-Agent On-Device Learning Approach
• Learning through Probing: a decentralized reinforcement learning architecture for social dilemmas
• Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning
• Group-Agent Reinforcement Learning
• Pommerman: A Multi-Agent Playground
• Distributed Policy Iteration for Scalable Approximation of Cooperative Multi-Agent Policies
• Class-Incremental Experience Replay for Continual Learning under Concept Drift
• Intelligent Link Adaptation for Grant-Free Access Cellular Networks: A Distributed Deep Reinforcement Learning Approach
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Active Perception in Adversarial Scenarios using Maximum Entropy Deep Reinforcement Learning
• Decentralized Multi-Agents by Imitation of a Centralized Controller
• Entropy based Independent Learning in Anonymous Multi-Agent Settings
• Information-Bottleneck-Based Behavior Representation Learning for Multi-agent Reinforcement learning
• Value Variance Minimization for Learning Approximate Equilibrium in Aggregation Systems
• Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control
• Logical Team Q-learning: An approach towards factored policies in cooperative MARL
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Voting-Based Multi-Agent Reinforcement Learning for Intelligent IoT