Evaluating Stochastic Rankings with Expected Exposure

Abstract

We introduce the concept of \emph{expected exposure} as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a \emph{distribution over rankings} instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including \emph{ad hoc} retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.

In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM '20). Association for Computing Machinery, New York, NY, USA

References in corpus (16)

• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Fairness of Exposure in Rankings
• FA*IR: A Fair Top-k Ranking Algorithm
• How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility
• Equity of Attention: Amortizing Individual Fairness in Rankings
• Fairness Without Demographics in Repeated Loss Minimization
• A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices
• Introducing LETOR 4.0 Datasets
• Multisided Fairness for Recommendation
• Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
• Auditing Search Engines for Differential Satisfaction Across Demographics
• LensKit for Python: Next-Generation Software for Recommender System Experiments
• Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs
• Policy Learning for Fairness in Ranking

Cited by in corpus (42)

• A Survey on the Fairness of Recommender Systems
• Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
• Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking
• Fairness of Exposure in Light of Incomplete Exposure Estimation
• Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank
• Understanding and Mitigating the Effect of Outliers in Fair Ranking
• Distributionally-Informed Recommender System Evaluation
• Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models
• A Probabilistic Position Bias Model for Short-Video Recommendation Feeds
• Evaluating Fairness in Argument Retrieval
• FAIR: Fairness-Aware Information Retrieval Evaluation
• Are We Really Achieving Better Beyond-Accuracy Performance in Next Basket Recommendation?
• Introducing the Expohedron for Efficient Pareto-optimal Fairness-Utility Amortizations in Repeated Rankings
• Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity
• Recommending With, Not For: Co-Designing Recommender Systems for Social Good
• Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking
• The Role of Relevance in Fair Ranking
• "We Need a Woman in Music": Exploring Wikipedia's Values on Article Priority
• Predictive Uncertainty-based Bias Mitigation in Ranking
• Properties of Group Fairness Metrics for Rankings
• Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance
• Pareto-Optimal Fairness-Utility Amortizations in Rankings with a DBN Exposure Model
• Comparing Fair Ranking Metrics
• Overview of the TREC 2019 Fair Ranking Track
• Patterns of gender-specializing query reformulation
• FARA: Future-aware Ranking Algorithm for Fairness Optimization
• Prompt-to-Slate: Diffusion Models for Prompt-Conditioned Slate Generation
• Learn to be Fair without Labels: a Distribution-based Learning Framework for Fair Ranking
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Fairness in Ranking under Uncertainty
• Language Fairness in Multilingual Information Retrieval
• A Reproducibility Study of Product-side Fairness in Bundle Recommendation
• Learning to Re-rank with Constrained Meta-Optimal Transport
• Incentives for Item Duplication under Fair Ranking Policies
• Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness
• Practical Compositional Fairness: Understanding Fairness in Multi-Component Recommender Systems
• Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees
• Exposure-Based Reinforcement Learning to Rank
• Searching Personal Collections
• Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier
• Robust Reputation Independence in Ranking Systems for Multiple Sensitive Attributes
• Estimation of Fair Ranking Metrics with Incomplete Judgments