Guided Deep Reinforcement Learning for Swarm Systems
arXiv:1709.06011
Abstract
In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally sensed information. In contrast, the critic is learned based on the true global state. Our algorithm uses deep reinforcement learning to approximate both the Q-function and the policy. The performance of the algorithm is evaluated on two tasks with simple simulated 2D agents: 1) finding and maintaining a certain distance to each others and 2) locating a target.
arXiv comment15 pages, 8 figures, accepted at the AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Workshop
References in corpus (5)
Cited by in corpus (21)
- Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
- QPLEX: Duplex Dueling Multi-Agent Q-Learning
- Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
- Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
- Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
- Autonomous Drone Swarm Navigation and Multi-target Tracking in 3D Environments with Dynamic Obstacles
- RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
- Clone Swarms: Learning to Predict and Control Multi-Robot Systems by Imitation
- Mis-spoke or mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning
- Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning
- Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
- Multi-agent navigation based on deep reinforcement learning and traditional pathfinding algorithm
- Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach
- Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph
- BGC: Multi-Agent Group Belief with Graph Clustering
- FireCommander: An Interactive, Probabilistic Multi-agent Environment for Heterogeneous Robot Teams
- Augmenting the action space with conventions to improve multi-agent cooperation in Hanabi
- Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
- Reinforcement Learning using Guided Observability