On-Policy Robot Imitation Learning from a Converging Supervisor

Abstract

Existing on-policy imitation learning algorithms, such as DAgger, assume access to a fixed supervisor. However, there are many settings where the supervisor may evolve during policy learning, such as a human performing a novel task or an improving algorithmic controller. We formalize imitation learning from a "converging supervisor" and provide sublinear static and dynamic regret guarantees against the best policy in hindsight with labels from the converged supervisor, even when labels during learning are only from intermediate supervisors. We then show that this framework is closely connected to a class of reinforcement learning (RL) algorithms known as dual policy iteration (DPI), which alternate between training a reactive learner with imitation learning and a model-based supervisor with data from the learner. Experiments suggest that when this framework is applied with the state-of-the-art deep model-based RL algorithm PETS as an improving supervisor, it outperforms deep RL baselines on continuous control tasks and provides up to an 80-fold speedup in policy evaluation.

Conference on Robot Learning (CoRL) 2019 Oral. First two authors contributed equally

References in corpus (11)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• One-Shot Visual Imitation Learning via Meta-Learning
• Model-Ensemble Trust-Region Policy Optimization
• Thinking Fast and Slow with Deep Learning and Tree Search
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Dual Policy Iteration
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• Online Learning with Continuous Variations: Dynamic Regret and Reductions

Cited by in corpus (10)

• VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Inverse Reinforcement Learning from a Gradient-based Learner
• LazyDAgger: Reducing Context Switching in Interactive Imitation Learning
• ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions
• Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
• Inverse Contextual Bandits: Learning How Behavior Evolves over Time