Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms

Abstract

We formulate the problem of learning to imitate multiple, non-deterministic teachers with minimal interaction cost. Rather than learning a specific policy as in standard imitation learning, the goal in this problem is to learn a distribution over a policy space. We first present a general framework that efficiently models and estimates such a distribution by learning continuous representations of the teacher policies. Next, we develop Active Performance-Based Imitation Learning (APIL), an active learning algorithm for reducing the learner-teacher interaction cost in this framework. By making query decisions based on predictions of future progress, our algorithm avoids the pitfalls of traditional uncertainty-based approaches in the face of teacher behavioral uncertainty. Results on both toy and photo-realistic navigation tasks show that APIL significantly reduces the numbers of interactions with teachers without compromising on performance. Moreover, it is robust to various degrees of teacher behavioral uncertainty.

References in corpus (8)

• On Evaluation of Embodied Navigation Agents
• Bayesian Active Learning for Classification and Preference Learning
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Search-based Structured Prediction
• Imitation Learning as $f$-Divergence Minimization
• Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables

Cited by in corpus (1)

• SAFARI: Safe and Active Robot Imitation Learning with Imagination