Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
arXiv:1907.05634
Abstract
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservatively-extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform significantly prior works in sample efficiency.
References in corpus (23)
- Distributed Representations of Words and Phrases and their Compositionality
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Deep Q-learning from Demonstrations
- Graph networks as learnable physics engines for inference and control
- Model-Ensemble Trust-Region Policy Optimization
- A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Reinforcement Learning from Imperfect Demonstrations
- Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
- Learning model-based planning from scratch
- DART: Noise Injection for Robust Imitation Learning
- Coordinated Multi-Agent Imitation Learning
- Dual Policy Iteration
- Generative predecessor models for sample-efficient imitation learning