Evaluating Stochastic Rankings with Expected Exposure

Abstract

We introduce the concept of \emph{expected exposure} as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a \emph{distribution over rankings} instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including \emph{ad hoc} retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.

In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM '20). Association for Computing Machinery, New York, NY, USA

References in corpus (6)

• BPR: Bayesian Personalized Ranking from Implicit Feedback
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Multisided Fairness for Recommendation
• Auditing Search Engines for Differential Satisfaction Across Demographics
• Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs
• Policy Learning for Fairness in Ranking

Cited by in corpus (41)

• A Survey on the Fairness of Recommender Systems
• Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
• Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking
• Fairness of Exposure in Light of Incomplete Exposure Estimation
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
• Understanding and Mitigating the Effect of Outliers in Fair Ranking
• Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models
• Distributionally-Informed Recommender System Evaluation
• Evaluating Fairness in Argument Retrieval
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• FAIR: Fairness-Aware Information Retrieval Evaluation
• Are We Really Achieving Better Beyond-Accuracy Performance in Next Basket Recommendation?
• Introducing the Expohedron for Efficient Pareto-optimal Fairness-Utility Amortizations in Repeated Rankings
• Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity
• Recommending With, Not For: Co-Designing Recommender Systems for Social Good
• The Role of Relevance in Fair Ranking
• Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking
• "We Need a Woman in Music": Exploring Wikipedia's Values on Article Priority
• Predictive Uncertainty-based Bias Mitigation in Ranking
• Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance
• Properties of Group Fairness Metrics for Rankings
• Overview of the TREC 2019 Fair Ranking Track
• Pareto-Optimal Fairness-Utility Amortizations in Rankings with a DBN Exposure Model
• Patterns of gender-specializing query reformulation
• Comparing Fair Ranking Metrics
• FARA: Future-aware Ranking Algorithm for Fairness Optimization
• Learn to be Fair without Labels: a Distribution-based Learning Framework for Fair Ranking
• Prompt-to-Slate: Diffusion Models for Prompt-Conditioned Slate Generation
• Language Fairness in Multilingual Information Retrieval
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Fairness in Ranking under Uncertainty
• Learning to Re-rank with Constrained Meta-Optimal Transport
• A Reproducibility Study of Product-side Fairness in Bundle Recommendation
• Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees
• Robust Reputation Independence in Ranking Systems for Multiple Sensitive Attributes
• Searching Personal Collections
• Estimation of Fair Ranking Metrics with Incomplete Judgments
• Incentives for Item Duplication under Fair Ranking Policies
• Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness
• Exposure-Based Reinforcement Learning to Rank
• Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier