Reinforcement Learning from Imperfect Demonstrations

Abstract

Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from the environment. These tasks have divergent losses which are difficult to jointly optimize and such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstrations and refines the policy in the environment, surpassing the demonstrator's performance. Crucially, both learning from demonstration and interactive refinement use the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

References in corpus (4)

• End to End Learning for Self-Driving Cars
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning

Cited by in corpus (53)

• Decision Transformer: Reinforcement Learning via Sequence Modeling
• PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Off-Policy Deep Reinforcement Learning without Exploration
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Projection-Based Constrained Policy Optimization
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Deep reinforcement learning for guidewire navigation in coronary artery phantom
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
• Goal-conditioned Imitation Learning
• Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations
• Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
• Backplay: "Man muss immer umkehren"
• Deep Reinforcement Learning based Local Planner for UAV Obstacle Avoidance using Demonstration Data
• Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning
• Constrained-Space Optimization and Reinforcement Learning for Complex Tasks
• AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers
• How You Act Tells a Lot: Privacy-Leakage Attack on Deep Reinforcement Learning
• Generalization Guarantees for Imitation Learning
• Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
• Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft
• Safe Reinforcement Learning with Natural Language Constraints
• Reinforcement Learning with Supervision from Noisy Demonstrations
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Policy Learning Using Weak Supervision
• Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Convergent and Efficient Deep Q Network Algorithm
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
• Reinforced Imitation in Heterogeneous Action Space
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• A Quantitative Perspective on Values of Domain Knowledge for Machine Learning
• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• Credit Assignment Safety Learning from Human Demonstrations
• Generative Inverse Deep Reinforcement Learning for Online Recommendation
• Co-Imitation Learning without Expert Demonstration
• Constrained Exploration and Recovery from Experience Shaping
• An Imitation Learning Approach to Unsupervised Parsing
• Diluted Near-Optimal Expert Demonstrations for Guiding Dialogue Stochastic Policy Optimisation
• Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learning
• Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
• Episodic Self-Imitation Learning with Hindsight
• Reinforcement Learning for Nested Polar Code Construction
• Instance-Aware Predictive Navigation in Multi-Agent Environments
• Meta Adaptation using Importance Weighted Demonstrations
• Multi-Preference Actor Critic
• End-to-end grasping policies for human-in-the-loop robots via deep reinforcement learning
• Mixing Human Demonstrations with Self-Exploration in Experience Replay for Deep Reinforcement Learning
• Evolutionary Stochastic Policy Distillation
• Towards intervention-centric causal reasoning in learning agents