From Optimizing Engagement to Measuring Value

Abstract

Most recommendation engines today are based on predicting user engagement, e.g. predicting whether a user will click on an item or not. However, there is potentially a large gap between engagement signals and a desired notion of "value" that is worth optimizing for. We use the framework of measurement theory to (a) confront the designer with a normative question about what the designer values, (b) provide a general latent variable model approach that can be used to operationalize the target construct and directly optimize for it, and (c) guide the designer in evaluating and revising their operationalization. We implement our approach on the Twitter platform on millions of users. In line with established approaches to assessing the validity of measurements, we perform a qualitative evaluation of how well our model captures a desired notion of "value".

Published at FAccT'21

References in corpus (2)

• A Practical Algorithm for Topic Modeling with Provable Guarantees
• On Measurement Bias in Causal Inference

Cited by in corpus (12)

• On the Opportunities and Risks of Foundation Models
• Building Human Values into Recommender Systems: An Interdisciplinary Synthesis
• Embedding Democratic Values into Social Media AIs via Societal Objective Functions
• Community notes reduce engagement with and diffusion of false information online
• Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints
• On component interactions in two-stage recommender systems
• Optimization's Neglected Normative Commitments
• The MineRL BASALT Competition on Learning from Human Feedback
• From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap
• Avoiding Tampering Incentives in Deep RL via Decoupled Approval
• Recommendation and Temptation
• Prediction Accuracy and Autonomy