Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning

Abstract

This work considers two distinct settings: imitation learning and goal-conditioned reinforcement learning. In either case, effective solutions require the agent to reliably reach a specified state (a goal), or set of states (a demonstration). Drawing a connection between probabilistic long-term dynamics and the desired value function, this work introduces an approach which utilizes recent advances in density estimation to effectively learn to reach a given state. As our first contribution, we use this approach for goal-conditioned reinforcement learning and show that it is both efficient and does not suffer from hindsight bias in stochastic domains. As our second contribution, we extend the approach to imitation learning and show that it achieves state-of-the art demonstration sample-efficiency on standard benchmark tasks.

References in corpus (11)

• Auto-Encoding Variational Bayes
• Pixel Recurrent Neural Networks
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
• Density estimation using Real NVP
• Hindsight Experience Replay
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
• Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation
• Generative predecessor models for sample-efficient imitation learning
• Is the Policy Gradient a Gradient?

Cited by in corpus (5)

• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• Outcome-Driven Reinforcement Learning via Variational Inference