Connecting Generative Adversarial Networks and Actor-Critic Methods

Abstract

Both generative adversarial networks (GAN) in unsupervised learning and actor-critic methods in reinforcement learning (RL) have gained a reputation for being difficult to optimize. Practitioners in both fields have amassed a large number of strategies to mitigate these instabilities and improve training. Here we show that GANs can be viewed as actor-critic methods in an environment where the actor cannot affect the reward. We review the strategies for stabilizing training for each class of models, both those that generalize between the two and those that are particular to that model. We also review a number of extensions to GANs and RL algorithms with even more complicated information flow. We hope that by highlighting this formal connection we will encourage both GAN and RL communities to develop general, scalable, and stable algorithms for multilevel optimization with deep networks, and to draw inspiration across communities.

Added comments on inverse reinforcement learning

References in corpus (11)

• Continuous control with deep reinforcement learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Energy-based Generative Adversarial Network
• Adversarial Feature Learning
• Adversarially Learned Inference
• Autoencoding beyond pixels using a learned similarity metric
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Generative Adversarial Imitation Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• On distinguishability criteria for estimating generative models

Cited by in corpus (52)

• NIPS 2016 Tutorial: Generative Adversarial Networks
• How Generative Adversarial Networks and Their Variants Work: An Overview
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• The Numerics of GANs
• The Mechanics of n-Player Differentiable Games
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions
• Feature-map-level Online Adversarial Knowledge Distillation
• Competitive Gradient Descent
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• On the Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Efficient Regret Minimization in Non-Convex Games
• TF-Replicator: Distributed Machine Learning for Researchers
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Zeroth-Order Algorithms for Nonconvex Minimax Problems with Improved Complexities
• Negative Momentum for Improved Game Dynamics
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• Statistical guarantees for generative models without domination
• Adversarially Guided Actor-Critic
• A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singleton
• Associative Adversarial Networks
• Enforcing constraints for time series prediction in supervised, unsupervised and reinforcement learning
• Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks
• ACtuAL: Actor-Critic Under Adversarial Learning
• Rates of convergence for density estimation with generative adversarial networks
• Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning
• Rethinking Exposure Bias In Language Modeling
• Poincaré Recurrence, Cycles and Spurious Equilibria in Gradient-Descent-Ascent for Non-Convex Non-Concave Zero-Sum Games
• A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization
• A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Games
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)
• ScarGAN: Chained Generative Adversarial Networks to Simulate Pathological Tissue on Cardiovascular MR Scans
• Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
• Stochastic Extragradient: General Analysis and Improved Rates
• Value-Function-based Sequential Minimization for Bi-level Optimization
• Stable Training of DNN for Speech Enhancement based on Perceptually-Motivated Black-Box Cost Function
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• Integrable Nonparametric Flows
• Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning
• Stateless actor-critic for instance segmentation with high-level priors
• Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning
• Parallelized Reverse Curriculum Generation
• Inspiration Learning through Preferences
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Complex Momentum for Optimization in Games
• Multi-Task Generative Adversarial Nets with Shared Memory for Cross-Domain Coordination Control
• Adversarial and Perceptual Refinement for Compressed Sensing MRI Reconstruction