Generative Adversarial Self-Imitation Learning

Abstract

This paper explores a simple regularizer for reinforcement learning by proposing Generative Adversarial Self-Imitation Learning (GASIL), which encourages the agent to imitate past good trajectories via generative adversarial imitation learning framework. Instead of directly maximizing rewards, GASIL focuses on reproducing past good trajectories, which can potentially make long-term credit assignment easier when rewards are sparse and delayed. GASIL can be easily combined with any policy gradient objective by using GASIL as a learned shaped reward function. Our experimental results show that GASIL improves the performance of proximal policy optimization on 2D Point Mass and MuJoCo environments with delayed reward and stochastic dynamics.

References in corpus (13)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Generative Adversarial Imitation Learning
• Model-Free Episodic Control
• Self-Imitation Learning
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Neural Program Synthesis with Priority Queue Training
• Learning Self-Imitating Diverse Policies
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning

Cited by in corpus (15)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Off-Policy Reinforcement Learning with Delayed Rewards
• Adversarial Imitation Learning via Random Search
• State-only Imitation with Transition Dynamics Mismatch
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Self-Imitation Advantage Learning
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Hindsight Generative Adversarial Imitation Learning
• A Max-Min Entropy Framework for Reinforcement Learning
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• Learning Guidance Rewards with Trajectory-space Smoothing
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• Learning Diverse Policies with Soft Self-Generated Guidance
• Randomized Adversarial Imitation Learning for Autonomous Driving