Estimating the Causal Impact of Recommendation Systems from Observational Data

Abstract

Recommendation systems are an increasingly prominent part of the web, accounting for up to a third of all traffic on several of the world's most popular sites. Nevertheless, little is known about how much activity such systems actually cause over and above activity that would have occurred via other means (e.g., search) if recommendations were absent. Although the ideal way to estimate the causal impact of recommendations is via randomized experiments, such experiments are costly and may inconvenience users. In this paper, therefore, we present a method for estimating causal effects from purely observational data. Specifically, we show that causal identification through an instrumental variable is possible when a product experiences an instantaneous shock in direct traffic and the products recommended next to it do not. We then apply our method to browsing logs containing anonymized activity for 2.1 million users on Amazon.com over a 9 month period and analyze over 4,000 unique products that experience such shocks. We find that although recommendation click-throughs do account for a large fraction of traffic among these products, at least 75% of this activity would likely occur in the absence of recommendations. We conclude with a discussion about the assumptions under which the method is appropriate and caveats around extrapolating results to other products, sites, or settings.

ACM Conference on Economics and Computation (EC 2015)

Cited by in corpus (20)

• How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility
• Unbiased Learning for the Causal Effect of Recommendation
• A survey on Adversarial Recommender Systems: from Attack/Defense strategies to Generative Adversarial Networks
• Evolution of Ego-networks in Social Media with Link Recommendations
• The Limits of Popularity-Based Recommendations, and the Role of Social Ties
• The long-term impact of ranking algorithms in growing networks
• Unleashing the Power of Large Language Model for Denoising Recommendation
• Contextual Multi-Armed Bandits for Causal Marketing
• The Importance of Causality in Decision Making: A Perspective on Recommender Systems
• Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering
• Shopping in the Multiverse: A Counterfactual Approach to In-Session Attribution
• When the Umpire is also a Player: Bias in Private Label Product Recommendations on E-commerce Marketplaces
• Correcting the User Feedback-Loop Bias for Recommendation Systems
• Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method
• Split-Treatment Analysis to Rank Heterogeneous Causal Effects for Prospective Interventions
• Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networks
• Online Evaluation Methods for the Causal Effect of Recommendations
• Identifying Causal Influences on Publication Trends and Behavior: A Case Study of the Computational Linguistics Community
• Estimating Attention Flow in Online Video Networks
• Causality-Aware Neighborhood Methods for Recommender Systems