Forward and inverse reinforcement learning sharing network weights and hyperparameters

Abstract

This paper proposes model-free imitation learning named Entropy-Regularized Imitation Learning (ERIL) that minimizes the reverse Kullback-Leibler (KL) divergence. ERIL combines forward and inverse reinforcement learning (RL) under the framework of an entropy-regularized Markov decision process. An inverse RL step computes the log-ratio between two distributions by evaluating two binary discriminators. The first discriminator distinguishes the state generated by the forward RL step from the expert's state. The second discriminator, which is structured by the theory of entropy regularization, distinguishes the state-action-next-state tuples generated by the learner from the expert ones. One notable feature is that the second discriminator shares hyperparameters with the forward RL, which can be used to control the discriminator's ability. A forward RL step minimizes the reverse KL estimated by the inverse RL step. We show that minimizing the reverse KL divergence is equivalent to finding an optimal policy. Our experimental results on MuJoCo-simulated environments and vision-based reaching tasks with a robotic arm show that ERIL is more sample-efficient than the baseline methods. We apply the method to human behaviors that perform a pole-balancing task and describe how the estimated reward functions show how every subject achieves her goal.

arXiv commentAccepted for publication in the Neural Networks

References in corpus (26)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Generative Adversarial Imitation Learning
• Population Based Training of Neural Networks
• From Variational to Deterministic Autoencoders
• Generative Adversarial Imitation from Observation
• DART: Noise Injection for Robust Imitation Learning
• Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Prescribed Generative Adversarial Networks
• Understanding the impact of entropy on policy optimization
• Imitation Learning as $f$-Divergence Minimization
• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
• TD-Regularized Actor-Critic Methods
• A Divergence Minimization Perspective on Imitation Learning Methods
• Sample-Efficient Imitation Learning via Generative Adversarial Nets
• Discount Factor as a Regularizer in Reinforcement Learning
• Entropic Regularization of Markov Decision Processes
• Augmenting GAIL with BC for sample efficient imitation learning
• Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learning
• Adversarial Imitation Learning from Incomplete Demonstrations
• Discriminator Soft Actor Critic without Extrinsic Rewards

Cited by in corpus (1)

• Model-Based Imitation Learning Using Entropy Regularization of Model and Policy