Graph Convolutional Reinforcement Learning

Abstract

Learning to cooperate is crucially important in multi-agent environments. The key is to understand the mutual interplay between agents. However, multi-agent environments are highly dynamic, where agents keep moving and their neighbors change quickly. This makes it hard to learn abstract representations of mutual interplay between agents. To tackle these difficulties, we propose graph convolutional reinforcement learning, where graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment, and relation kernels capture the interplay between agents by their relation representations. Latent features produced by convolutional layers from gradually increased receptive fields are exploited to learn cooperation, and cooperation is further improved by temporal relation regularization for consistency. Empirically, we show that our method substantially outperforms existing methods in a variety of cooperative scenarios.

arXiv commentICLR'20

References in corpus (14)

• Deep Convolutional Networks on Graph-Structured Data
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Interaction Networks for Learning about Objects, Relations and Physics
• Counterfactual Multi-Agent Policy Gradients
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Learning Attentional Communication for Multi-Agent Cooperation
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Relational Deep Reinforcement Learning
• Visual Interaction Networks
• Deep Multi-Agent Reinforcement Learning with Relevance Graphs
• MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence
• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning

Cited by in corpus (16)

• Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning
• Graph Policy Gradients for Large Scale Robot Control
• Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication
• Multi-Agent Game Abstraction via Graph Attention Neural Network
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic with Hierarchical Graph Attention Network
• Where are the Keys? -- Learning Object-Centric Navigation Policies on Semantic Maps with Graph Convolutional Networks
• When2com: Multi-Agent Perception via Communication Graph Grouping
• Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems
• Joint Attention for Multi-Agent Coordination and Social Learning
• Collaborative Graph Walk for Semi-supervised Multi-Label Node Classification
• Who2com: Collaborative Perception via Learnable Handshake Communication
• Foresight of Graph Reinforcement Learning Latent Permutations Learnt by Gumbel Sinkhorn Network
• AB-Mapper: Attention and BicNet Based Multi-agent Path Finding for Dynamic Crowded Environment
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph