Provable Representation Learning for Imitation Learning via Bi-level Optimization

Abstract

A common strategy in modern learning systems is to learn a representation that is useful for many tasks, a.k.a. representation learning. We study this strategy in the imitation learning setting for Markov decision processes (MDPs) where multiple experts' trajectories are available. We formulate representation learning as a bi-level optimization problem where the "outer" optimization tries to learn the joint representation and the "inner" optimization encodes the imitation learning setup and tries to learn task-specific parameters. We instantiate this framework for the imitation learning settings of behavior cloning and observation-alone. Theoretically, we show using our framework that representation learning can provide sample complexity benefits for imitation learning in both settings. We also provide proof-of-concept experiments to verify our theory.

26 pages

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• One-Shot Visual Imitation Learning via Meta-Learning
• Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
• Meta-Learning with Implicit Gradients
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• The Benefit of Multitask Representation Learning
• Online Meta-Learning
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Learning to Search Better Than Your Teacher
• Imitating Latent Policies from Observation
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
• Provably Efficient Imitation Learning from Observation Alone

Cited by in corpus (3)

• Toward the Fundamental Limits of Imitation Learning
• Near-optimal Representation Learning for Linear Bandits and Linear RL
• On the Power of Multitask Representation Learning in Linear MDP