Entropy Regularization for Mean Field Games with Learning
arXiv:2010.00145
Abstract
Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.
References in corpus (14)
- Continuous control with deep reinforcement learning
- VIME: Variational Information Maximizing Exploration
- Parameter Space Noise for Exploration
- Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
- Mean field games via controlled martingale problems: Existence of Markovian equilibria
- A unified view of entropy-regularized Markov decision processes
- Provably Efficient Maximum Entropy Exploration
- Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games
- Modeling Others using Oneself in Multi-Agent Reinforcement Learning
- Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods
- Exploration versus exploitation in reinforcement learning: a stochastic control approach
- Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
- Information-Theoretic Confidence Bounds for Reinforcement Learning
- Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time