Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions

Abstract

We consider the problem of local planning in fixed-horizon and discounted Markov Decision Processes (MDPs) with linear function approximation and a generative model under the assumption that the optimal action-value function lies in the span of a feature map that is available to the planner. Previous work has left open the question of whether there exist sound planners that need only poly(H,d) queries regardless of the MDP, where H is the horizon and d is the dimensionality of the features. We answer this question in the negative: we show that any sound planner must query at least $\min (\exp (\Omega (d)),\Omega (2^H))$ samples in the fized-horizon setting and $\exp (\Omega (d))$ samples in the discounted setting. We also show that for any $\delta >0$, the least-squares value iteration algorithm with $O(H^5{d}^{H+1}/{\delta }^2)$ queries can compute a $\delta$-optimal policy in the fixed-horizon setting. We discuss implications and remaining open questions.

References in corpus (5)

• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Provably Efficient $Q$-learning with Function Approximation via Distribution Shift Error Checking Oracle
• Efficient Planning in Large MDPs with Weak Linear Function Approximation

Cited by in corpus (15)

• Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
• Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processes
• Logarithmic Regret for Reinforcement Learning with Linear Function Approximation
• Bilinear Classes: A Structural Framework for Provable Generalization in RL
• Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximation
• An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap
• The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
• Fast Rates for the Regret of Offline Reinforcement Learning
• On Query-efficient Planning in MDPs under Linear Realizability of the Optimal State-value Function
• Efficient Local Planning with Linear Function Approximation
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation
• Gap-Dependent Bounds for Two-Player Markov Games
• Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDP