Provable Fictitious Play for General Mean-Field Games
arXiv:2010.04211
Abstract
We propose a reinforcement learning algorithm for stationary mean-field games, where the goal is to learn a pair of mean-field state and stationary policy that constitutes the Nash equilibrium. When viewing the mean-field state and the policy as two players, we propose a fictitious play algorithm which alternatively updates the mean-field state and the policy via gradient-descent and proximal policy optimization, respectively. Our algorithm is in stark contrast with previous literature which solves each single-agent reinforcement learning problem induced by the iterates mean-field states to the optimum. Furthermore, we prove that our fictitious play algorithm converges to the Nash equilibrium at a sublinear rate. To the best of our knowledge, this seems the first provably convergent single-loop reinforcement learning algorithm for mean-field games based on iterative updates of both mean-field state and policy.
References in corpus (18)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Dota 2 with Large Scale Deep Reinforcement Learning
- Multi-agent Reinforcement Learning in Sequential Social Dilemmas
- Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
- A Distributional Perspective on Reinforcement Learning
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity
- A Theory of Regularized Markov Decision Processes
- Learning from Distributions via Support Measure Machines
- Emergent Communication through Negotiation
- Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
- Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
- Mean Field Games with Ergodic cost for Discrete Time Markov Processes
- Learning to Collaborate in Markov Decision Processes
- Dynamic Regret of Policy Optimization in Non-stationary Environments
- Learning in Discounted-cost and Average-cost Mean-field Games
- Discrete-time Risk-sensitive Mean-field Games