Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies

Abstract

This paper investigates constrained multi-agent reinforcement learning (CMARL) in coupled environments, where agents collaboratively maximize the sum of local objectives while satisfying individual safety constraints. Existing studies face two limitations: (1) most rely on independent policies that fail to capture complex interactions in coupled environments; and (2) agents require the global Lagrange multipliers, which are sensitive learned variables whose global sharing risks exposing private agent-specific information. To overcome these issues, we propose a framework where agents adopt coupled policies that depend on both the states and policy parameters of their $\hat{I}{º}_p$-hop neighbors, where $\hat{I}{º}_p>0$ denotes the coupling distance, and develop a distributed and scalable primal-dual (DSPD) algorithm wherein each agent accesses only information within a prescribed local neighborhood. In the proposed algorithm, agents exchange sensitive parameters only with immediate neighbors over a separate time-varying network, while maintaining local estimates to execute the coupled policy. We establish that the proposed algorithm achieves $\hat{I}\mu$-first-order stationary convergence with approximation error $\mathcal{O}(γ^{\frac{κ+1}{κ_{p}}})$, where $\hat{I}º>0$ is the truncated distance and ${\hat{I}}^3\in (0,1)$ is discount factor. Simulations on a wireless access-control network demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms, validating its effectiveness.