My new working paper with Eric Schwartz, “The A/B Test Deception: Divergent Delivery, Ad Response Heterogeneity, and Erroneous Inferences in Online Advertising Field Experiments” is now available.

You can access the paper and a companion app at the project website or from [SSRN]

How do you know which of two versions of your ad is better?” Online platforms like Facebook giver advertisers tools to conduct “randomized” experiments to test which ad works best. But that’s not what the experiment is testing if the platform is targeting users based on the content of the ads. Instead, the comparison is a jumble of the effectiveness of ad content, and how well that particular platform puts your ad in front of the right users. Advertisers who want to isolate the effect of the creative content of the ads are out of luck.

There’s also a companion [web app] (https://braunm.github.io/ABTestDeception) also available for download as an R package.