Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation

Abstract

Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.

References in corpus (10)

• A Brief Survey of Deep Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Representation Learning via Invariant Causal Mechanisms
• Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
• Deep Reinforcement Learning from Policy-Dependent Human Feedback
• The EMPATHIC Framework for Task Learning from Implicit Human Feedback
• FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback
• Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning

Cited by in corpus (3)

• Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?
• Towards Interactive Reinforcement Learning with Intrinsic Feedback
• Explainability via Responsibility