Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Survey

Abstract

The majority of multi-agent system (MAS) implementations aim to optimise agents' policies with respect to a single objective, despite the fact that many real-world problem domains are inherently multi-objective in nature. Multi-objective multi-agent systems (MOMAS) explicitly consider the possible trade-offs between conflicting objective functions. We argue that, in MOMAS, such compromises should be analysed on the basis of the utility that these compromises have for the users of a system. As is standard in multi-objective optimisation, we model the user utility using utility functions that map value or return vectors to scalar values. This approach naturally leads to two different optimisation criteria: expected scalarised returns (ESR) and scalarised expected returns (SER). We develop a new taxonomy which classifies multi-objective multi-agent decision making settings, on the basis of the reward structures, and which and how utility functions are applied. This allows us to offer a structured view of the field, to clearly delineate the current state-of-the-art in multi-objective multi-agent decision making approaches and to identify promising directions for future research. Starting from the execution phase, in which the selected policies are applied and the utility for the users is attained, we analyse which solution concepts apply to the different settings in our taxonomy. Furthermore, we define and discuss these solution concepts under both ESR and SER optimisation criteria. We conclude with a summary of our main findings and a discussion of many promising future research directions in multi-objective multi-agent systems.

Under review since 15 May 2019

References in corpus (24)

• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• StarCraft II: A New Challenge for Reinforcement Learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• A Survey of Multi-Objective Sequential Decision-Making
• Counterfactual Multi-Agent Policy Gradients
• Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Optimal and Approximate Q-value Functions for Decentralized POMDPs
• A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
• A Multi-Objective Deep Reinforcement Learning Framework
• Multi-Agent Generative Adversarial Imitation Learning
• Opponent Modeling in Deep Reinforcement Learning
• Multi-objective Stackelberg Game Between a Regulating Authority and a Mining Company: A Case Study in Environmental Economics
• Collective Intelligence for Control of Distributed Dynamical Systems
• Multi-Objective Deep Reinforcement Learning
• Emergent Coordination Through Competition
• Actor-Critic Policy Optimization in Partially Observable Multiagent Environments
• Structure in the Value Function of Two-Player Zero-Sum Games of Incomplete Information
• Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning
• Approximation of Lorenz-Optimal Solutions in Multiobjective Markov Decision Processes

Cited by in corpus (10)

• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• A utility-based analysis of equilibria in multi-objective normal form games
• Deep Multi-Objective Reinforcement Learning for Utility-Based Infrastructural Maintenance Optimization
• Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Making
• Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search
• A survey of air combat behavior modeling using machine learning
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Exploring the Impact of Tunable Agents in Sequential Social Dilemmas
• All unconstrained strongly convex problems are weakly simplicial
• A Survey of Requirements for COVID-19 Mitigation Strategies. Part II: Elicitation of Requirements