MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
arXiv:2102.11448
Abstract
In many contemporary applications such as healthcare, finance, robotics, and recommendation systems, continuous deployment of new policies for data collection and online learning is either cost ineffective or impractical. We consider a setting that lies between pure offline reinforcement learning (RL) and pure online RL called deployment constrained RL in which the number of policy deployments for data sampling is limited. To solve this challenging task, we propose a new algorithmic learning framework called Model-based Uncertainty regularized and Sample Efficient Batch Optimization (MUSBO). Our framework discovers novel and high quality samples for each deployment to enable efficient data collection. During each offline training session, we bootstrap the policy update by quantifying the amount of uncertainty within our collected data. In the high support region (low uncertainty), we encourage our policy by taking an aggressive update. In the low support region (high uncertainty) when the policy bootstraps into the out-of-distribution region, we downweight it by our estimated uncertainty quantification. Experimental results show that MUSBO achieves state-of-the-art performance in the deployment constrained RL setting.
References in corpus (21)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Solving Rubik's Cube with a Robot Hand
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Model-Ensemble Trust-Region Policy Optimization
- Behavior Regularized Offline Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Benchmarking Model-Based Reinforcement Learning
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- When to Trust Your Model: Model-Based Policy Optimization
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
- Entity Abstraction in Visual Model-Based Reinforcement Learning
- Provably Efficient Q-Learning with Low Switching Cost
- Prediction and Control with Temporal Segment Models