Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints

Abstract

Inverse reinforcement learning (IRL) enables an agent to learn complex behavior by observing demonstrations from a (near-)optimal policy. The typical assumption is that the learner's goal is to match the teacher's demonstrated behavior. In this paper, we consider the setting where the learner has its own preferences that it additionally takes into consideration. These preferences can for example capture behavioral biases, mismatched worldviews, or physical constraints. We study two teaching approaches: learner-agnostic teaching, where the teacher provides demonstrations from an optimal policy ignoring the learner's preferences, and learner-aware teaching, where the teacher accounts for the learner's preferences. We design learner-aware teaching algorithms and show that significant performance improvements can be achieved over learner-agnostic teaching.

References in corpus (7)

• Cooperative Inverse Reinforcement Learning
• An Overview of Machine Teaching
• Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners
• Teaching Inverse Reinforcement Learners via Features and Demonstrations
• Repeated Inverse Reinforcement Learning
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• Lifelong Inverse Reinforcement Learning

Cited by in corpus (8)

• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• The Teaching Dimension of Kernel Perceptron
• Classification Under Human Assistance
• Iterative Teaching by Label Synthesis
• Teaching an Active Learner with Contrastive Examples
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
• Regression Under Human Assistance