Information Theoretic Model Predictive Q-Learning
arXiv:2001.02153
Abstract
Model-free Reinforcement Learning (RL) works well when experience can be collected cheaply and model-based RL is effective when system dynamics can be modeled accurately. However, both assumptions can be violated in real world problems such as robotics, where querying the system can be expensive and real-world dynamics can be difficult to model. In contrast to RL, Model Predictive Control (MPC) algorithms use a simulator to optimize a simple policy class online, constructing a closed-loop controller that can effectively contend with real-world dynamics. MPC performance is usually limited by factors such as model bias and the limited horizon of optimization. In this work, we present a novel theoretical connection between information theoretic MPC and entropy regularized RL and develop a Q-learning algorithm that can leverage biased models. We validate the proposed algorithm on sim-to-sim control tasks to demonstrate the improvements over optimal control and reinforcement learning from scratch. Our approach paves the way for deploying reinforcement learning algorithms on real systems in a systematic manner.
arXiv commentExtended version (15 pages) of paper accepted at the 2nd Learning for Dynamics and Control (L4DC) Conference, 2020
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- DeepMind Control Suite
- Reinforcement Learning with Deep Energy-Based Policies
- Model-Ensemble Trust-Region Policy Optimization
- Thinking Fast and Slow with Deep Learning and Tree Search
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Equivalence Between Policy Gradients and Soft Q-Learning
- Model-Based Active Exploration
- Taming the Noise in Reinforcement Learning via Soft Updates
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control