Generative Adversarial Imitation Learning

Abstract

Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert's cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Model-Free Imitation Learning with Policy Optimization

Cited by in corpus (151)

• Spectral Normalization for Generative Adversarial Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Deep Reinforcement Learning: An Overview
• Deep reinforcement learning from human preferences
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• An Algorithmic Perspective on Imitation Learning
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Robust Motion In-betweening
• One-Shot Imitation Learning
• Maximum a Posteriori Policy Optimisation
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Third-Person Imitation Learning
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Deep Learning in Mobile and Wireless Networking: A Survey
• Scalable agent alignment via reward modeling: a research direction
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Universal Planning Networks
• Active Learning in Robotics: A Review of Control Principles
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• Machine Theory of Mind
• Geometry Score: A Method For Comparing Generative Adversarial Networks
• Robust Imitation of Diverse Behaviors
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Neural Text Generation: Past, Present and Beyond
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• Generative Adversarial Self-Imitation Learning
• Reward learning from human preferences and demonstrations in Atari
• Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
• Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations
• Learning to Defend by Learning to Attack
• A General Language Assistant as a Laboratory for Alignment
• Observational Learning by Reinforcement Learning
• Global Convergence to the Equilibrium of GANs using Variational Inequalities
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
• The Atari Grand Challenge Dataset
• Paraphrase Generation with Deep Reinforcement Learning
• Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning
• Mutual Alignment Transfer Learning
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Towards a Better Understanding and Regularization of GAN Training Dynamics
• Robot eye-hand coordination learning by watching human demonstrations: a task function approximation approach
• Error Bounds of Imitating Policies and Environments
• What Matters for Adversarial Imitation Learning?
• Self-Supervised Reinforcement Learning for Recommender Systems
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Multi-task Maximum Entropy Inverse Reinforcement Learning
• UniCon: Universal Neural Controller For Physics-based Character Motion
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• Event Extraction with Generative Adversarial Imitation Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Learning Norms from Stories: A Prior for Value Aligned Agents
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• Hindsight Trust Region Policy Optimization
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Energy-Based Sequence GANs for Recommendation and Their Connection to Imitation Learning
• Reinforcement Learning with Supervision from Noisy Demonstrations
• Vision-based Robot Manipulation Learning via Human Demonstrations
• Adversarial Imitation Learning from Incomplete Demonstrations
• Active Reinforcement Learning with Monte-Carlo Tree Search
• Visual Forecasting by Imitating Dynamics in Natural Sequences
• Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Density Matching Reward Learning
• Imitating Driver Behavior with Generative Adversarial Networks
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
• KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge
• Efficient Model-Free Reinforcement Learning Using Gaussian Process
• Zero-shot Task Adaptation using Natural Language
• Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control
• Understanding Human Behaviors in Crowds by Imitating the Decision-Making Process
• Guaranteeing Reproducibility in Deep Learning Competitions
• Off-Policy Adversarial Inverse Reinforcement Learning
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors
• A Geometric Perspective on Visual Imitation Learning
• Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• MobILE: Model-Based Imitation Learning From Observation Alone
• Imitation Learning with Sinkhorn Distances
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped
• Defining the problem of Observation Learning
• Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
• Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
• On Value Discrepancy of Imitation Learning
• Hybrid Reinforcement Learning with Expert State Sequences
• Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning
• Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
• Maximum Entropy Multi-Task Inverse RL
• Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples
• Discriminator Soft Actor Critic without Extrinsic Rewards
• Inverse Reinforcement Learning with Missing Data
• Incremental Adversarial Learning for Optimal Path Planning
• Imperfect also Deserves Reward: Multi-Level and Sequential Reward Modeling for Better Dialog Management
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching
• Constrained Exploration and Recovery from Experience Shaping
• Adversarial Learning of Task-Oriented Neural Dialog Models
• Autonomy 2.0: Why is self-driving always 5 years away?
• Imitation Learning for Fashion Style Based on Hierarchical Multimodal Representation
• Sufficiently Accurate Model Learning
• Amanuensis: The Programmer's Apprentice
• Imitating Unknown Policies via Exploration
• Learning to Optimize via Wasserstein Deep Inverse Optimal Control
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Motion Reasoning for Goal-Based Imitation Learning
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Reward is enough for convex MDPs
• Manipulate as Human: Learning Task-oriented Manipulation Skills by Adversarial Motion Priors
• Generative Adversarial Imitation Learning for Empathy-based AI
• Classification of sparsely labeled spatio-temporal data through semi-supervised adversarial learning
• Toward Imitating Visual Attention of Experts in Software Development Tasks
• Learning without Knowing: Unobserved Context in Continuous Transfer Reinforcement Learning
• Transferring Agent Behaviors from Videos via Motion GANs
• Inspiration Learning through Preferences
• Optimal wireless rate and power control in the presence of jammers using reinforcement learning
• Cost Functions for Robot Motion Style
• Dyna-AIL : Adversarial Imitation Learning by Planning
• Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration
• Reducing the Deployment-Time Inference Control Costs of Deep Reinforcement Learning Agents via an Asymmetric Architecture
• Mitigation of Adversarial Policy Imitation via Constrained Randomization of Policy (CRoP)
• Quickest change detection with unknown parameters: Constant complexity and near optimality
• Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories
• Mature GAIL: Imitation Learning for Low-level and High-dimensional Input using Global Encoder and Cost Transformation
• RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learning
• Variational Reward Estimator Bottleneck: Learning Robust Reward Estimator for Multi-Domain Task-Oriented Dialog
• Characterizing Policy Divergence for Personalized Meta-Reinforcement Learning
• Scenario Generalization of Data-driven Imitation Models in Crowd Simulation
• Towards intervention-centric causal reasoning in learning agents
• The Past and Present of Imitation Learning: A Citation Chain Study
• Deep Reinforcement Learning for High Level Character Control