Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
arXiv:2006.14804
Abstract
Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.
References in corpus (17)
- A Brief Survey of Deep Reinforcement Learning
- Dueling Network Architectures for Deep Reinforcement Learning
- Generative Adversarial Imitation Learning
- Deep Q-learning from Demonstrations
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- Interactive Learning from Policy-Dependent Human Feedback
- Deep Attention Recurrent Q-Network
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback
- Representation Learning via Invariant Causal Mechanisms
- Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
- Deep Reinforcement Learning from Policy-Dependent Human Feedback
- The EMPATHIC Framework for Task Learning from Implicit Human Feedback
- Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset
- FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback
- Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning