MARL with General Utilities via Decentralized Shadow Reward Actor-Critic

Abstract

We posit a new mechanism for cooperation in multi-agent reinforcement learning (MARL) based upon any nonlinear function of the team's long-term state-action occupancy measure, i.e., a \emph{general utility}. This subsumes the cumulative return but also allows one to incorporate risk-sensitivity, exploration, and priors. % We derive the {\bf D}ecentralized {\bf S}hadow Reward {\bf A}ctor-{\bf C}ritic (DSAC) in which agents alternate between policy evaluation (critic), weighted averaging with neighbors (information mixing), and local gradient updates for their policy parameters (actor). DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i.e., the "shadow reward". DSAC converges to $\epsilon$-stationarity in $\mathcal{O}(1/{\epsilon }^{2.5})$ (Theorem \ref{theorem:final}) or faster $\mathcal{O}(1/{\epsilon }^2)$ (Corollary \ref{corollary:communication}) steps with high probability, depending on the amount of communications. We further establish the non-existence of spurious stationary points for this problem, that is, DSAC finds the globally optimal policy (Corollary \ref{corollary:global}). Experiments demonstrate the merits of goals beyond the cumulative return in cooperative MARL.

References in corpus (14)

• Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
• Deep Reinforcement Learning for Page-wise Recommendations
• Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• Provably Efficient Maximum Entropy Exploration
• Coordinated Multi-Agent Imitation Learning
• Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning
• Biases for Emergent Communication in Multi-agent Reinforcement Learning
• Active Exploration in Markov Decision Processes
• Cautious Reinforcement Learning via Distributional Risk in the Dual Domain
• Primal-Dual Distributed Temporal Difference Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• Correcting Experience Replay for Multi-Agent Communication