Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

Abstract

The potential benefits of model-free reinforcement learning to real robotics systems are limited by its uninformed exploration that leads to slow convergence, lack of data-efficiency, and unnecessary interactions with the environment. To address these drawbacks we propose a method that combines reinforcement and imitation learning by shaping the reward function with a state-and-action-dependent potential that is trained from demonstration data, using a generative model. We show that this accelerates policy learning by specifying high-value areas of the state and action space that are worth exploring first. Unlike the majority of existing methods that assume optimal demonstrations and incorporate the demonstration data as hard constraints on policy optimization, we instead incorporate demonstration data as advice in the form of a reward shaping potential trained as a generative model of states and actions. In particular, we examine both normalizing flows and Generative Adversarial Networks to represent these potentials. We show that, unlike many existing approaches that incorporate demonstrations as hard constraints, our approach is unbiased even in the case of suboptimal and noisy demonstrations. We present an extensive range of simulations, as well as experiments on the Franka Emika 7DOF arm, to demonstrate the practicality of our method.

submitted to ICRA 2021

References in corpus (16)

• End to End Learning for Self-Driving Cars
• Improved Training of Wasserstein GANs
• Variational Inference with Normalizing Flows
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Deep Reinforcement Learning that Matters
• Deep Q-learning from Demonstrations
• Behavior Regularized Offline Reinforcement Learning
• Reinforcement Learning from Imperfect Demonstrations
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Fast Policy Learning through Imitation and Reinforcement
• Residual Policy Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• Reinforcement Learning with Supervision from Noisy Demonstrations

Cited by in corpus (2)

• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Tolerance-Guided Policy Learning for Adaptable and Transferrable Delicate Industrial Insertion