Algorithm or Creative? Disentangling Algorithmic Divergent Delivery from User Ad Preferences in Digital Platforms

Abstract

Online advertising platforms host hundreds of thousands of A/B tests, but the platform's delivery algorithm routes each creative to the audience it predicts will engage. Every two-arm test therefore conflates the creative's effect with the algorithm's targeting response, and adjusting for the realized audience is biased because audience is a post-treatment mediator. We propose a three-arm design that adds an arm exposing the algorithm to the treatment metadata while holding the user-facing creative identical to control, separately point-identifying the algorithmic delivery and ad creative effects. In a live Meta campaign with a women-targeted text fragment, the algorithmic delivery channel raises female impression share by $+2.27$ ppt, while the visible creative leaves delivery composition statistically unchanged. A conventional two-arm test would attribute the entire delivery shift to the creative. The design isolates the contribution of platform's algorithm to the outcome which is separable from creative content.