MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning

Abstract

Deep reinforcement learning (RL) has been applied extensively to solve complex decision-making problems. In many real-world scenarios, tasks often have several conflicting objectives and may require multiple agents to cooperate, which are the multi-objective multi-agent decision-making problems. However, only few works have been conducted on this intersection. Existing approaches are limited to separate fields and can only handle multi-agent decision-making with a single objective, or multi-objective decision-making with a single agent. In this paper, we propose MO-MIX to solve the multi-objective multi-agent reinforcement learning (MOMARL) problem. Our approach is based on the centralized training with decentralized execution (CTDE) framework. A weight vector representing preference over the objectives is fed into the decentralized agent network as a condition for local action-value function estimation, while a mixing network with parallel architecture is used to estimate the joint action-value function. In addition, an exploration guide approach is applied to improve the uniformity of the final non-dominated solutions. Experiments demonstrate that the proposed method can effectively solve the multi-objective multi-agent cooperative decision-making problem and generate an approximation of the Pareto set. Our approach not only significantly outperforms the baseline method in all four kinds of evaluation metrics, but also requires less computational cost.

15 pages, 10 figures, published in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

References in corpus (16)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Trust Region Policy Optimization
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Deep Recurrent Q-Learning for Partially Observable MDPs
• A Survey of Multi-Objective Sequential Decision-Making
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Optimal and Approximate Q-value Functions for Decentralized POMDPs
• Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation
• Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
• Multi-Objective Deep Reinforcement Learning
• A Review of Cooperative Multi-Agent Deep Reinforcement Learning
• Dynamic Weights in Multi-Objective Deep Reinforcement Learning
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Policy Regularization via Noisy Advantage Values for Cooperative Multi-agent Actor-Critic methods