On the Fairness of Randomized Trials for Recommendation with Heterogeneous Demographics and Beyond

Abstract

Observed events in recommendation are consequence of the decisions made by a policy, thus they are usually selectively labeled, namely the data are Missing Not At Random (MNAR), which often causes large bias to the estimate of true outcomes risk. A general approach to correct MNAR bias is performing small Randomized Controlled Trials (RCTs), where an additional uniform policy is employed to randomly assign items to each user. In this work, we concentrate on the fairness of RCTs under both homogeneous and heterogeneous demographics, especially analyzing the bias for the least favorable group on the latter setting. Considering RCTs' limitations, we propose a novel Counterfactual Robust Risk Minimization (CRRM) framework, which is totally free of expensive RCTs, and derive its theoretical generalization error bound. At last, empirical experiments are performed on synthetic tasks and real-world data sets, substantiating our method's superiority both in fairness and generalization.

The latest version of this paper has been published in NeurIPS'20 with the name: Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback. Please refer to arXiv:2009.02623

References in corpus (7)

• Equality of Opportunity in Supervised Learning
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Doubly Robust Policy Evaluation and Learning
• Fairness Without Demographics in Repeated Loss Minimization
• Policy Learning for Fairness in Ranking
• Average Individual Fairness: Algorithms, Generalization and Experiments
• Missing Not at Random in Matrix Completion: The Effectiveness of Estimating Missingness Probabilities Under a Low Nuclear Norm Assumption