Graph Policy Gradients for Large Scale Robot Control

Abstract

In this paper, we consider the problem of learning policies to control a large number of homogeneous robots. To this end, we propose a new algorithm we call Graph Policy Gradients (GPG) that exploits the underlying graph symmetry among the robots. The curse of dimensionality one encounters when working with a large number of robots is mitigated by employing a graph convolutional neural (GCN) network to parametrize policies for the robots. The GCN reduces the dimensionality of the problem by learning filters that aggregate information among robots locally, similar to how a convolutional neural network is able to learn local features in an image. Through experiments on formation flying, we show that our proposed method is able to scale better than existing reinforcement methods that employ fully connected networks. More importantly, we show that by using our locally learned filters we are able to zero-shot transfer policies trained on just three robots to over hundred robots.

References in corpus (11)

• Semi-Supervised Classification with Graph Convolutional Networks
• Playing Atari with Deep Reinforcement Learning
• End-to-End Training of Deep Visuomotor Policies
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Counterfactual Multi-Agent Policy Gradients
• Convolutional Neural Network Architectures for Signals Supported on Graphs
• Stability Properties of Graph Neural Networks
• Graph Convolutional Reinforcement Learning
• TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow
• Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients
• Concurrent Meta Reinforcement Learning

Cited by in corpus (7)

• Graph Neural Networks for Motion Planning
• Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning
• Where are the Keys? -- Learning Object-Centric Navigation Policies on Semantic Maps with Graph Convolutional Networks
• Graph Policy Gradients for Large Scale Unlabeled Motion Planning with Constraints
• Represented Value Function Approach for Large Scale Multi Agent Reinforcement Learning
• Inner Attention Supported Adaptive Cooperation for Heterogeneous Multi Robots Teaming based on Multi-agent Reinforcement Learning
• Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph