Teaching Inverse Reinforcement Learners via Features and Demonstrations

Abstract

Learning near-optimal behaviour from an expert's demonstrations typically relies on the assumption that the learner knows the features that the true reward function depends on. In this paper, we study the problem of learning from demonstrations in the setting where this is not the case, i.e., where there is a mismatch between the worldviews of the learner and the expert. We introduce a natural quantity, the teaching risk, which measures the potential suboptimality of policies that look optimal to the learner in this setting. We show that bounds on the teaching risk guarantee that the learner is able to find a near-optimal policy using standard algorithms based on inverse reinforcement learning. Based on these findings, we suggest a teaching scheme in which the expert can decrease the teaching risk by updating the learner's worldview, and thus ultimately enable her to find a near-optimal policy.

arXiv commentNeurIPS'2018 (extended version)

Cited by in corpus (23)

• Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
• Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
• Feature Expansive Reward Learning: Rethinking Human Input
• Reinforcement Learning for Education: Opportunities and Challenges
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• Policy Teaching in Reinforcement Learning via Environment Poisoning Attacks
• Teaching Multiple Concepts to a Forgetful Learner
• Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners
• Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• Curriculum Design for Teaching via Demonstrations: Theory and Applications
• The Teaching Dimension of Kernel Perceptron
• Fair Clustering Using Antidote Data
• Reinforcement Learning Under Algorithmic Triage
• Teaching to Learn: Sequential Teaching of Agents with Inner States
• The Sample Complexity of Teaching-by-Reinforcement on Q-Learning
• Classification Under Human Assistance
• Teaching an Active Learner with Contrastive Examples
• Iterative Teaching by Label Synthesis
• Preference-Based Batch and Sequential Teaching
• Environment Shaping in Reinforcement Learning using State Abstraction
• Class Teaching for Inverse Reinforcement Learners
• Regression Under Human Assistance