Supervised Off-Policy Ranking

Abstract

Off-policy evaluation (OPE) is to evaluate a target policy with data generated by other policies. Most previous OPE methods focus on precisely estimating the true performance of a policy. We observe that in many applications, (1) the end goal of OPE is to compare two or multiple candidate policies and choose a good one, which is a much simpler task than precisely evaluating their true performance; and (2) there are usually multiple policies that have been deployed to serve users in real-world systems and thus the true performance of these policies can be known. Inspired by the two observations, in this work, we study a new problem, supervised off-policy ranking (SOPR), which aims to rank a set of target policies based on supervised learning by leveraging off-policy data and policies with known performance. We propose a method to solve SOPR, which learns a policy scoring model by minimizing a ranking loss of the training policies rather than estimating the precise policy performance. The scoring model in our method, a hierarchical Transformer based model, maps a set of state-action pairs to a score, where the state of each pair comes from the off-policy data and the action is taken by a target policy on the state in an offline manner. Extensive experiments on public datasets show that our method outperforms baseline methods in terms of rank correlation, regret value, and stability. Our code is publicly available at GitHub.

arXiv commentAccepted by ICML 2022

References in corpus (21)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Doubly Robust Policy Evaluation and Learning
• Attention, Learn to Solve Routing Problems!
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Critic Regularized Regression
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• GenDICE: Generalized Offline Estimation of Stationary Values
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Off-Policy Evaluation via Off-Policy Classification
• Off-Policy Evaluation via the Regularized Lagrangian
• Q($\lambda$) with Off-Policy Corrections
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization