Corruption-Robust Offline Reinforcement Learning

Abstract

We study the adversarial robustness in offline reinforcement learning. Given a batch dataset consisting of tuples $(s,a,r,s^{\prime })$, an adversary is allowed to arbitrarily modify $\epsilon$ fraction of the tuples. From the corrupted dataset the learner aims to robustly identify a near-optimal policy. We first show that a worst-case $\Omega (d\epsilon )$ optimality gap is unavoidable in linear MDP of dimension $d$, even if the adversary only corrupts the reward element in a tuple. This contrasts with dimension-free results in robust supervised learning and best-known lower-bound in the online RL setting with corruption. Next, we propose robust variants of the Least-Square Value Iteration (LSVI) algorithm utilizing robust supervised learning oracles, which achieve near-matching performances in cases both with and without full data coverage. The algorithm requires the knowledge of $\epsilon$ to design the pessimism bonus in the no-coverage case. Surprisingly, in this case, the knowledge of $\epsilon$ is necessary, as we show that being adaptive to unknown $\epsilon$ is impossible.This again contrasts with recent results on corruption-robust online RL and implies that robust offline RL is a strictly harder problem.

References in corpus (24)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• MOReL : Model-Based Offline Reinforcement Learning
• Sever: A Robust Meta-Algorithm for Stochastic Optimization
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Recent Advances in Algorithmic High-Dimensional Robust Statistics
• Safe Policy Improvement with Baseline Bootstrapping
• COMBO: Conservative Offline Model-Based Policy Optimization
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Sample-Optimal Parametric Q-Learning Using Linearly Additive Features
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
• Robust regression with covariate filtering: Heavy tails and adversarial contamination
• Robust Policy Gradient against Strong Data Corruption
• Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximation
• Improved Corruption Robust Algorithms for Episodic Reinforcement Learning

Cited by in corpus (3)

• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Representation Learning for Online and Offline RL in Low-rank MDPs
• A Model Selection Approach for Corruption Robust Reinforcement Learning