PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes
arXiv:2006.06352
Abstract
We consider the problem of batch multi-task reinforcement learning with observed context descriptors, motivated by its application to personalized medical treatment. In particular, we study two general classes of learning algorithms: direct policy learning (DPL), an imitation-learning based approach which learns from expert trajectories, and model-based learning. First, we derive sample complexity bounds for DPL, and then show that model-based learning from expert actions can, even with a finite model class, be impossible. After relaxing the conditions under which the model-based approach is expected to learn by allowing for greater coverage of state-action space, we provide sample complexity bounds for model-based learning with finite model classes, showing that there exist model classes with sample complexity exponential in their statistical complexity. We then derive a sample complexity upper bound for model-based learning based on a measure of concentration of the data distribution. Our results give formal justification for imitation learning over model-based learning in this setting.
References in corpus (9)
- Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
- Dynamics-Aware Unsupervised Discovery of Skills
- Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
- Sharing Knowledge in Multi-Task Deep Reinforcement Learning
- Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
- Transfer from Multiple MDPs
- Single Episode Policy Transfer in Reinforcement Learning
- No-regret Exploration in Contextual Reinforcement Learning
- Markov Decision Processes with Continuous Side Information