Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning

Abstract

This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is then extended to a multi-agent variant of the proximal policy optimization algorithm and demonstrated on particle and multiwalker robot environments that have characteristics such as system health, risk-taking, semi-expendable agents, continuous action spaces, and partial observability. We show significant improvement in learning performance compared to policy gradient methods that do not perform multi-agent credit assignment.

References in corpus (9)

• Trust Region Policy Optimization
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• MAVEN: Multi-Agent Variational Exploration
• Clipped Action Policy Gradient

Cited by in corpus (1)

• Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning