RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states

Abstract

Multi-agent deep reinforcement learning makes optimal decisions dependent on system states observed by agents, but any uncertainty on the observations may mislead agents to take wrong actions. The Mean-Field Actor-Critic reinforcement learning (MFAC) is well-known in the multi-agent field since it can effectively handle a scalability problem. However, it is sensitive to state perturbations that can significantly degrade the team rewards. This work proposes a Robust Mean-field Actor-Critic reinforcement learning (RoMFAC) that has two innovations: 1) a new objective function of training actors, composed of a \emph{policy gradient function} that is related to the expected cumulative discount reward on sampled clean states and an \emph{action loss function} that represents the difference between actions taken on clean and adversarial states; and 2) a repetitive regularization of the action loss, ensuring the trained actors to obtain excellent performance. Furthermore, this work proposes a game model named a State-Adversarial Stochastic Game (SASG). Despite the Nash equilibrium of SASG may not exist, adversarial perturbations to states in the RoMFAC are proven to be defensible based on SASG. Experimental results show that RoMFAC is robust against adversarial perturbations while maintaining its competitive performance in environments without perturbations.

References in corpus (6)

• Explaining and Harnessing Adversarial Examples
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• A Comprehensive Survey of Scene Graphs: Generation and Application
• Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems
• Sim-to-Real Transfer of Accurate Grasping with Eye-In-Hand Observations and Continuous Control
• Sparse Adversarial Attack in Multi-agent Reinforcement Learning

Cited by in corpus (4)

• Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges
• Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention
• Enhancing the Robustness of QMIX against State-adversarial Attacks
• Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents