Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters

Abstract

In this work, we argue for the importance of an online evaluation budget for a reliable comparison of deep offline RL algorithms. First, we delineate that the online evaluation budget is problem-dependent, where some problems allow for less but others for more. And second, we demonstrate that the preference between algorithms is budget-dependent across a diverse range of decision-making domains such as Robotics, Finance, and Energy Management. Following the points above, we suggest reporting the performance of deep offline RL algorithms under varying online evaluation budgets. To facilitate this, we propose to use a reporting tool from the NLP field, Expected Validation Performance. This technique makes it possible to reliably estimate expected maximum performance under different budgets while not requiring any additional computation beyond hyperparameter search. By employing this tool, we also show that Behavioral Cloning is often more favorable to offline RL algorithms when working within a limited budget.

arXiv commentICML 2022, Spotlight; https://tinkoff-ai.github.io/eop/

References in corpus (19)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Deep Reinforcement Learning that Matters
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Critic Regularized Regression
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• A Benchmark Environment Motivated by Industrial Control Problems
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Off-Policy Evaluation via Off-Policy Classification
• Show Your Work: Improved Reporting of Experimental Results
• Offline RL Without Off-Policy Evaluation
• Showing Your Work Doesn't Always Work