Fast Policy Learning through Imitation and Reinforcement

Abstract

Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.

Cited by in corpus (12)

• Off-Policy Deep Reinforcement Learning without Exploration
• Riemannian Motion Policy Fusion through Learnable Lyapunov Function Reshaping
• Imitation-Projected Programmatic Reinforcement Learning
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Interactive-Predictive Neural Machine Translation through Reinforcement and Imitation
• Agile Autonomous Driving using End-to-End Deep Imitation Learning
• Robotic Imitation of Human Assembly Skills Using Hybrid Trajectory and Force Learning
• Co-training for Policy Learning
• Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods
• EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints
• Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning