Control as Hybrid Inference
arXiv:2007.05838
Abstract
The field of reinforcement learning can be split into model-based and model-free methods. Here, we unify these approaches by casting model-free policy optimisation as amortised variational inference, and model-based planning as iterative variational inference, within a `control as hybrid inference' (CHI) framework. We present an implementation of CHI which naturally mediates the balance between iterative and amortised inference. Using a didactic experiment, we demonstrate that the proposed algorithm operates in a model-based manner at the onset of learning, before converging to a model-free algorithm once sufficient data have been collected. We verify the scalability of our algorithm on a continuous control benchmark, demonstrating that it outperforms strong model-free and model-based baselines. CHI thus provides a principled framework for harnessing the sample efficiency of model-based planning while retaining the asymptotic performance of model-free policy optimisation.
References in corpus (16)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Continuous Deep Q-Learning with Model-based Acceleration
- Model-Ensemble Trust-Region Policy Optimization
- Maximum a Posteriori Policy Optimisation
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Inference Suboptimality in Variational Autoencoders
- Exploring Model-based Planning with Policy Networks
- Variational Inference MPC for Bayesian Model-based Reinforcement Learning
- Iterative Amortized Inference
- Semi-Amortized Variational Autoencoders
- If MaxEnt RL is the Answer, What is the Question?
- On the challenges of learning with inference networks on sparse, high-dimensional data
- Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control
- A General Method for Amortizing Variational Filtering
- Robot Playing Kendama with Model-Based and Model-Free Reinforcement Learning