Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising

Abstract

Algorithm audits are powerful tools for studying black-box systems. While very effective in examining technical components, the method stops short of a sociotechnical frame, which would also consider users as an integral and dynamic part of the system. Addressing this gap, we propose the concept of sociotechnical auditing: auditing methods that evaluate algorithmic systems at the sociotechnical level, focusing on the interplay between algorithms and users as each impacts the other. Just as algorithm audits probe an algorithm with varied inputs and observe outputs, a sociotechnical audit (STA) additionally probes users, exposing them to different algorithmic behavior and measuring resulting attitudes and behaviors. To instantiate this method, we develop Intervenr, a platform for conducting browser-based, longitudinal sociotechnical audits with consenting, compensated participants. Intervenr investigates the algorithmic content users encounter online and coordinates systematic client-side interventions to understand how users change in response. As a case study, we deploy Intervenr in a two-week sociotechnical audit of online advertising (N=244) to investigate the central premise that personalized ad targeting is more effective on users. In the first week, we collect all browser ads delivered to users, and in the second, we deploy an ablation-style intervention that disrupts normal targeting by randomly pairing participants and swapping all their ads. We collect user-oriented metrics (self-reported ad interest and feeling of representation) and advertiser-oriented metrics (ad views, clicks, and recognition) throughout, along with a total of over 500,000 ads. Our STA finds that targeted ads indeed perform better with users, but also that users begin to acclimate to different ads in only a week, casting doubt on the primacy of personalized ad targeting given the impact of repeated exposure.

arXiv commentTo appear at CSCW 2023

References in corpus (11)

• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Discrimination through optimization: How Facebook's ad delivery can lead to skewed outcomes
• Social Influence in Social Advertising: Evidence from Field Experiments
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem
• Ablation Studies in Artificial Neural Networks
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• Auditing for Discrimination in Algorithms Delivering Job Ads
• 'I Just Want to Hack Myself to Not Get Distracted': Evaluating Design Interventions for Self-Control on Facebook
• Endorsements on Social Media: An Empirical Study of Affiliate Marketing Disclosures on YouTube and Pinterest
• Algorithms that "Don't See Color": Comparing Biases in Lookalike and Special Ad Audiences
• Searching for Representation: A sociotechnical audit of googling for members of U.S. Congress

Cited by in corpus (11)

• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
• Youth as Peer Auditors: Engaging Teenagers with Algorithm Auditing of Machine Learning Applications
• Lower Quantity, Higher Quality: Auditing News Content and User Perceptions on Twitter/X Algorithmic versus Chronological Timelines
• Access Denied: Meaningful Data Access for Quantitative Algorithm Audits
• Auditing for Racial Discrimination in the Delivery of Education Ads
• Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers
• Learning AI Auditing: A Case Study of Teenagers Auditing a Generative AI Model
• Navigating Automated Hiring: Perceptions, Strategy Use, and Outcomes Among Young Job Seekers
• External Evaluation of Discrimination Mitigation Efforts in Meta's Ad Delivery
• Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software
• On the Use of Proxies in Political Ad Targeting