Model Based Planning with Energy Based Models
arXiv:1909.06878
Abstract
Model-based planning holds great promise for improving both sample efficiency and generalization in reinforcement learning (RL). We show that energy-based models (EBMs) are a promising class of models to use for model-based planning. EBMs naturally support inference of intermediate states given start and goal state distributions. We provide an online algorithm to train EBMs while interacting with the environment, and show that EBMs allow for significantly better online learning than corresponding feed-forward networks. We further show that EBMs support maximum entropy state inference and are able to generate diverse state space plans. We show that inference purely in state space - without planning actions - allows for better generalization to previously unseen obstacles in the environment and prevents the planner from exploiting the dynamics model by applying uncharacteristic action sequences. Finally, we show that online EBM training naturally leads to intentionally planned state exploration which performs significantly better than random exploration.
arXiv commentCoRL 2019
References in corpus (12)
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- VIME: Variational Information Maximizing Exploration
- Reinforcement Learning with Deep Energy-Based Policies
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
- Learning Dexterous In-Hand Manipulation
- Implicit Generation and Generalization in Energy-Based Models
- Gotta Learn Fast: A New Benchmark for Generalization in RL
- Deep Directed Generative Models with Energy-Based Probability Estimation
- Exponential Family Estimation via Adversarial Dynamics Embedding
- Task-Agnostic Dynamics Priors for Deep Reinforcement Learning