Imitation with Neural Density Models
arXiv:2010.09808
Abstract
We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kullback-Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-art demonstration efficiency on benchmark control tasks.
References in corpus (28)
- Adam: A Method for Stochastic Optimization
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Spectral Normalization for Generative Adversarial Networks
- NICE: Non-linear Independent Components Estimation
- Improved Techniques for Training GANs
- Generative Modeling by Estimating Gradients of the Data Distribution
- Density estimation using Real NVP
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Reinforcement Learning with Deep Energy-Based Policies
- MADE: Masked Autoencoder for Distribution Estimation
- Exploration by Random Network Distillation
- Learning in Implicit Generative Models
- Multi-Agent Generative Adversarial Imitation Learning
- Implicit Generation and Generalization in Energy-Based Models
- RNADE: The real-valued neural autoregressive density-estimator
- Provably Efficient Maximum Entropy Exploration
- Efficient Exploration via State Marginal Matching
- Variational Inference using Implicit Distributions
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation
- What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
- SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
- Prescribed Generative Adversarial Networks
- Imitation Learning as -Divergence Minimization
- Imitation Learning via Off-Policy Distribution Matching
- Domain Adaptive Imitation Learning
- Training Deep Energy-Based Models with f-Divergence Minimization
- Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods
- Flow Contrastive Estimation of Energy-Based Models