Pessimistic Model Selection for Offline Deep Reinforcement Learning

Abstract

Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. Despite its promising performance, practical gaps exist when deploying DRL in real-world scenarios. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. In particular, for offline DRL with observational data, model selection is a challenging task as there is no ground truth available for performance demonstration, in contrast with the online setting with simulated environments. In this work, we propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee, which features a provably effective framework for finding the best policy among a set of candidate models. Two refined approaches are also proposed to address the potential bias of DRL model in identifying the optimal policy. Numerical studies demonstrated the superior performance of our approach over existing methods.

arXiv commentPreprint. A non-archival and preliminary venue was presented at NeurIPS 2021 Offline Reinforcement Learning Workshop

References in corpus (26)

• Playing Atari with Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Unity: A General Platform for Intelligent Agents
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Distributed Distributional Deterministic Policy Gradients
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy
• Distributional Reinforcement Learning with Quantile Regression
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• Exact Post Model Selection Inference for Marginal Screening
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Batch Policy Learning in Average Reward Markov Decision Processes
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policies
• Adaptive Estimator Selection for Off-Policy Evaluation
• Offline Policy Selection under Uncertainty