Curriculum Design for Teaching via Demonstrations: Theory and Applications
arXiv:2106.04696
Abstract
We consider the problem of teaching via demonstrations in sequential decision-making settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher's optimal policy and the learner's current policy. Compared to the state of the art, our strategy doesn't require access to the learner's internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore, we adapt our curriculum strategy to the setting where no teacher agent is present using task-specific difficulty scores. Experiments on a synthetic car driving environment and navigation-based environments demonstrate the effectiveness of our curriculum strategy.
arXiv commentNeurIPS 2021
References in corpus (14)
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Learning to Execute
- Maximum Entropy Deep Inverse Reinforcement Learning
- Learning by Playing - Solving Sparse Reward Tasks from Scratch
- Attention, Learn to Solve Routing Problems!
- Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks
- Iterative Machine Teaching
- Teaching Inverse Reinforcement Learners via Features and Demonstrations
- Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners
- Mix&Match - Agent Curricula for Reinforcement Learning
- Repeated Inverse Reinforcement Learning
- Preference-Based Batch and Sequential Teaching: Towards a Unified View of Models
- Optimal Cooperative Inference
- Dynamic Teaching in Sequential Decision Making Environments