Follow the Neurally-Perturbed Leader for Adversarial Training
arXiv:2002.06476
Abstract
Game-theoretic models of learning are a powerful set of models that optimize multi-objective architectures. Among these models are zero-sum architectures that have inspired adversarial learning frameworks. An important shortcoming of these zeros-sum architectures is that gradient-based training leads to weak convergence and cyclic dynamics. We propose a novel follow the leader training algorithm for zeros-sum architectures that guarantees convergence to mixed Nash equilibrium without cyclic behaviors. It is a special type of follow the perturbed leader algorithm where perturbations are the result of a neural mediating agent. We validate our theoretical results by applying this training algorithm to games with convex and non-convex loss as well as generative adversarial architectures. Moreover, we customize the implementation of this algorithm for adversarial imitation learning applications. At every step of the training, the mediator agent perturbs the observations with generated codes. As a result of these mediating codes, the proposed algorithm is also efficient for learning in environments with various factors of variations. We validate our assertion by using a procedurally generated game environment as well as synthetic data. Github implementation is available.
References in corpus (10)
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
- FeUdal Networks for Hierarchical Reinforcement Learning
- On Convergence and Stability of GANs
- Quantifying Generalization in Reinforcement Learning
- On Variational Bounds of Mutual Information
- The Mechanics of n-Player Differentiable Games
- Optimization, Learning, and Games with Predictable Sequences
- Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
- Efficient Regret Minimization in Non-Convex Games
- Hierarchical Soft Actor-Critic: Adversarial Exploration via Mutual Information Optimization