Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction

Abstract

Researchers have demonstrated state-of-the-art performance in sequential decision making problems (e.g., robotics control, sequential prediction) with deep neural network models. One often has access to near-optimal oracles that achieve good performance on the task during training. We demonstrate that AggreVaTeD --- a policy gradient extension of the Imitation Learning (IL) approach of (Ross & Bagnell, 2014) --- can leverage such an oracle to achieve faster and better solutions with less training data than a less-informed Reinforcement Learning (RL) technique. Using both feedforward and recurrent neural network predictors, we present stochastic gradient procedures on a sequential prediction task, dependency-parsing from raw image data, as well as on various high dimensional robotics control problems. We also provide a comprehensive theoretical study of IL that demonstrates we can expect up to exponentially lower sample complexity for learning with AggreVaTeD than with RL algorithms, which backs our empirical findings. Our results and theory indicate that the proposed approach can achieve superior performance with respect to the oracle when the demonstrator is sub-optimal.

17 pages

References in corpus (8)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Trust Region Policy Optimization
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Search-based Structured Prediction
• Learning Policies for Contextual Submodular Prediction

Cited by in corpus (36)

• An Algorithmic Perspective on Imitation Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Certified Adversarial Robustness for Deep Reinforcement Learning
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Active Deep Q-learning with Demonstration
• On-Policy Robot Imitation Learning from a Converging Supervisor
• Toward the Fundamental Limits of Imitation Learning
• On the Sample Complexity of Stability Constrained Imitation Learning
• Learning Dexterous Manipulation from Suboptimal Experts
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• Imitation Learning: Progress, Taxonomies and Challenges
• Imitation-Projected Programmatic Reinforcement Learning
• Online Learning with Continuous Variations: Dynamic Regret and Reductions
• Imitation Learning with Stability and Safety Guarantees
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning
• Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
• Robust Asymmetric Learning in POMDPs
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• Learning by Cheating
• Co-training for Policy Learning
• Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
• Continuous Online Learning and New Insights to Online Imitation Learning
• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
• Bootstrapping Motor Skill Learning with Motion Planning
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• Active Hierarchical Imitation and Reinforcement Learning
• Integration of Imitation Learning using GAIL and Reinforcement Learning using Task-achievement Rewards via Probabilistic Graphical Model
• Dynamic Regret Convergence Analysis and an Adaptive Regularization Algorithm for On-Policy Robot Imitation Learning
• Meta Adaptation using Importance Weighted Demonstrations
• Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories
• Ranking Policy Gradient