Large-Scale Multi-Agent Deep FBSDEs

Abstract

In this paper we present a scalable deep learning framework for finding Markovian Nash Equilibria in multi-agent stochastic games using fictitious play. The motivation is inspired by theoretical analysis of Forward Backward Stochastic Differential Equations (FBSDE) and their implementation in a deep learning setting, which is the source of our algorithm's sample efficiency improvement. By taking advantage of the permutation-invariant property of agents in symmetric games, the scalability and performance is further enhanced significantly. We showcase superior performance of our framework over the state-of-the-art deep fictitious play algorithm on an inter-bank lending/borrowing problem in terms of multiple metrics. More importantly, our approach scales up to 3000 agents in simulation, a scale which, to the best of our knowledge, represents a new state-of-the-art. We also demonstrate the applicability of our framework in robotics on a belief space autonomous racing problem.

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Solving high-dimensional partial differential equations using deep learning
• AutoAugment: Learning Augmentation Policies from Data
• Noisy Networks for Exploration
• Adversarially Robust Generalization Requires More Data
• Convergence of the Deep BSDE Method for Coupled FBSDEs
• MAVEN: Multi-Agent Variational Exploration
• Learning Optimal Control via Forward and Backward Stochastic Differential Equations
• Learning Deep Stochastic Optimal Control Policies using Forward-Backward SDEs
• Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Games
• On the Benefits of Invariance in Neural Networks
• Deep Fictitious Play for Stochastic Differential Games
• Deep 2FBSDEs For Systems With Control Multiplicative Noise
• Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning
• Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets