A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models

Abstract

Generative adversarial networks (GANs) are a recently proposed class of generative models in which a generator is trained to optimize a cost function that is being simultaneously learned by a discriminator. While the idea of learning cost functions is relatively new to the field of generative modeling, learning costs has long been studied in control and reinforcement learning (RL) domains, typically for imitation learning from demonstrations. In these fields, learning cost function underlying observed behavior is known as inverse reinforcement learning (IRL) or inverse optimal control. While at first the connection between cost learning in RL and cost learning in generative modeling may appear to be a superficial one, we show in this paper that certain IRL methods are in fact mathematically equivalent to GANs. In particular, we demonstrate an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator's density can be evaluated and is provided as an additional input to the discriminator. Interestingly, maximum entropy IRL is a special case of an energy-based model. We discuss the interpretation of GANs as an algorithm for training energy-based models, and relate this interpretation to other recent work that seeks to connect GANs and EBMs. By formally highlighting the connection between GANs, IRL, and EBMs, we hope that researchers in all three communities can better identify and apply transferable ideas from one domain to another, particularly for developing more stable and scalable algorithms: a major challenge in all three domains.

NIPS 2016 Workshop on Adversarial Training. First two authors contributed equally

References in corpus (13)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Energy-based Generative Adversarial Network
• Density estimation using Real NVP
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Generative Adversarial Imitation Learning
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• An Actor-Critic Algorithm for Sequence Prediction
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Data Generation as Sequential Decision Making
• Model-Free Imitation Learning with Policy Optimization

Cited by in corpus (106)

• NIPS 2016 Tutorial: Generative Adversarial Networks
• An Algorithmic Perspective on Imitation Learning
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Deep Learning and Knowledge-Based Methods for Computer Aided Molecular Design -- Toward a Unified Approach: State-of-the-Art and Future Directions
• Implicit Generation and Generalization in Energy-Based Models
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• A deep inverse reinforcement learning approach to route choice modeling with context-dependent rewards
• Stabilizing Generative Adversarial Networks: A Survey
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
• Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving
• Imitating Interactive Intelligence
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Face Super-Resolution Through Wasserstein GANs
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Learning Loss Functions for Semi-supervised Learning via Discriminative Adversarial Networks
• A Divergence Minimization Perspective on Imitation Learning Methods
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Paraphrase Generation with Deep Reinforcement Learning
• Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
• Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning
• SafeCritic: Collision-Aware Trajectory Prediction
• Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach
• Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Deep Bayesian Reward Learning from Preferences
• Energy-Based Imitation Learning
• $f$-GAIL: Learning $f$-Divergence for Generative Adversarial Imitation Learning
• State-only Imitation with Transition Dynamics Mismatch
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Weakly-supervised Knowledge Graph Alignment with Adversarial Learning
• Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• Energy-Based Sequence GANs for Recommendation and Their Connection to Imitation Learning
• Improving GAN Training via Binarized Representation Entropy (BRE) Regularization
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
• Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning
• Implicit Policy for Reinforcement Learning
• Non-Adversarial Imitation Learning and its Connections to Adversarial Methods
• Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning
• Adversarial Imitation via Variational Inverse Reinforcement Learning
• Trajectory Prediction with Latent Belief Energy-Based Model
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Unsupervised Perceptual Rewards for Imitation Learning
• Identifiability in inverse reinforcement learning
• Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Visual Adversarial Imitation Learning using Variational Models
• Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• A Pragmatic Look at Deep Imitation Learning
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Learning Latent Space Energy-Based Prior Model
• Robust Maximum Entropy Behavior Cloning
• Flow Contrastive Estimation of Energy-Based Models
• Domain-Robust Visual Imitation Learning with Mutual Information Constraints
• VILD: Variational Imitation Learning with Diverse-quality Demonstrations
• To Relieve Your Headache of Training an MRF, Take AdVIL
• Off-Policy Adversarial Inverse Reinforcement Learning
• Adversarial recovery of agent rewards from latent spaces of the limit order book
• Environment Reconstruction with Hidden Confounders for Reinforcement Learning based Recommendation
• Enhanced Experience Replay Generation for Efficient Reinforcement Learning
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• Deep Consensus Learning
• Regularized Inverse Reinforcement Learning
• Data Driven Aircraft Trajectory Prediction with Deep Imitation Learning
• A Relation Analysis of Markov Decision Process Frameworks
• Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
• Incremental Adversarial Learning for Optimal Path Planning
• Learning a Decision Module by Imitating Driver's Control Behaviors
• Discriminator Soft Actor Critic without Extrinsic Rewards
• Plan-Space State Embeddings for Improved Reinforcement Learning
• Learning Compact Reward for Image Captioning
• oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
• Learning Variable Impedance Control via Inverse Reinforcement Learning for Force-Related Tasks
• Support-weighted Adversarial Imitation Learning
• Reward function shape exploration in adversarial imitation learning: an empirical study
• Imitation Learning for Fashion Style Based on Hierarchical Multimodal Representation
• Sample Efficient Social Navigation Using Inverse Reinforcement Learning
• Curriculum Offline Imitation Learning
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Learning to Optimize via Wasserstein Deep Inverse Optimal Control
• Task Transfer by Preference-Based Cost Learning
• Scenario Generalization of Data-driven Imitation Models in Crowd Simulation
• CAMEO: Curiosity Augmented Metropolis for Exploratory Optimal Policies
• Learning Data-Driven Objectives to Optimize Interactive Systems
• Generative adversarial training of product of policies for robust and adaptive movement primitives
• Generalized Inverse Planning: Learning Lifted non-Markovian Utility for Generalizable Task Representation
• Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain
• Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts
• Stochastic Inverse Reinforcement Learning
• Pre-Training Transformers as Energy-Based Cloze Models
• An Inattention Model for Traveler Behavior with e-Coupons
• Sequential Anomaly Detection using Inverse Reinforcement Learning