Reparameterized Variational Divergence Minimization for Stable Imitation
arXiv:2006.10810
Abstract
While recent state-of-the-art results for adversarial imitation-learning algorithms are encouraging, recent works exploring the imitation learning from observation (ILO) setting, where trajectories \textit{only} contain expert observations, have not been met with the same success. Inspired by recent investigations of -divergence manipulation for the standard imitation learning setting(Ke et al., 2019; Ghasemipour et al., 2019), we here examine the extent to which variations in the choice of probabilistic divergence may yield more performant ILO algorithms. We unfortunately find that -divergence minimization through reinforcement learning is susceptible to numerical instabilities. We contribute a reparameterization trick for adversarial imitation learning to alleviate the optimization challenges of the promising -divergence minimization framework. Empirically, we demonstrate that our design choices allow for ILO algorithms that outperform baseline approaches and more closely match expert performance in low-dimensional continuous-control tasks.
References in corpus (28)
- End to End Learning for Self-Driving Cars
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Energy-based Generative Adversarial Network
- Which Training Methods for GANs do actually Converge?
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Generative Moment Matching Networks
- Deep Reinforcement Learning that Matters
- Deep Q-learning from Demonstrations
- Towards Automatic Learning of Procedures from Web Instructional Videos
- A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
- Lyapunov-based Safe Policy Optimization for Continuous Control
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
- A unified view of entropy-regularized Markov decision processes
- On integral probability metrics, ϕ-divergences and binary classification
- Generative Adversarial Imitation from Observation
- Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
- Learning Montezuma's Revenge from a Single Demonstration
- DART: Noise Injection for Robust Imitation Learning
- Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
- Time-Contrastive Networks: Self-Supervised Learning from Video
- Imitation Learning as -Divergence Minimization
- Hierarchical Imitation and Reinforcement Learning
- A Divergence Minimization Perspective on Imitation Learning Methods
- Provably Efficient Imitation Learning from Observation Alone
- Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
- Adversarially Robust Training through Structured Gradient Regularization
- Internal Model from Observations for Reward Shaping