Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems

Abstract

Recent successes combine reinforcement learning algorithms and deep neural networks, despite reinforcement learning not being widely applied to robotics and real world scenarios. This can be attributed to the fact that current state-of-the-art, end-to-end reinforcement learning approaches still require thousands or millions of data samples to converge to a satisfactory policy and are subject to catastrophic failures during training. Conversely, in real world scenarios and after just a few data samples, humans are able to either provide demonstrations of the task, intervene to prevent catastrophic actions, or simply evaluate if the policy is performing correctly. This research investigates how to integrate these human interaction modalities to the reinforcement learning loop, increasing sample efficiency and enabling real-time reinforcement learning in robotics and real world scenarios. This novel theoretical foundation is called Cycle-of-Learning, a reference to how different human interaction modalities, namely, task demonstration, intervention, and evaluation, are cycled and combined to reinforcement learning algorithms. Results presented in this work show that the reward signal that is learned based upon human interaction accelerates the rate of learning of reinforcement learning algorithms and that learning from a combination of human demonstrations and interventions is faster and more sample efficient when compared to traditional supervised learning algorithms. Finally, Cycle-of-Learning develops an effective transition between policies learned using human demonstrations and interventions to reinforcement learning. The theoretical foundation developed by this research opens new research paths to human-agent teaming scenarios where autonomous agents are able to learn from human teammates and adapt to mission performance metrics in real-time and in real world scenarios.

arXiv commentPhD thesis, Aerospace Engineering, Texas A&M (2020). For more information, see https://vggoecks.com/

References in corpus (18)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• End to End Learning for Self-Driving Cars
• Dueling Network Architectures for Deep Reinforcement Learning
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• StarCraft II: A New Challenge for Reinforcement Learning
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Recurrent World Models Facilitate Policy Evolution
• Continuous Deep Q-Learning with Model-based Acceleration
• Deep Q-learning from Demonstrations
• Maximum Entropy Deep Inverse Reinforcement Learning
• Continuous Inverse Optimal Control with Locally Optimal Examples
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Interactive Learning from Policy-Dependent Human Feedback
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• Cycle-of-Learning for Autonomous Systems from Human Interaction