Your Ad Test Compared Audiences, Not Just Ads

What they were looking at

An A/B test on an ad platform looks like a clean experiment. You’ve got two versions of an ad, the same budget, the same targeting settings, and a winner at the end.

So what’s the catch? Michael Braun and Eric Schwartz, two marketing professors, pointed at the step most advertisers never see. The platform’s algorithm decides who sees each version, and it learns as it goes. If Ad A starts getting clicks from one kind of person and Ad B from another, the algorithm goes and finds more of each. They call that divergent delivery. They extended the standard model of cause and effect to account for it, ran simulations, and then showed it happening in a real A/B test on a major ad platform.

What they found

“experimenters may not be learning what they think they are learning from ad A/B tests.” Braun & Schwartz (2025), abstract

Because the two ads end up in front of different audiences, the test can’t separate what the ad did from who it was shown to:

“the estimated comparisons confound the effect of the ad content with the effect of algorithmic targeting.” Same

And the damage goes well past a little noise:

“algorithmic targeting, user heterogeneity, and data aggregation conspire to confound the magnitude, and even the sign, of ad A/B test results.” Same

When the paper came out, Braun put it in plain terms: “The winning ad may have performed better simply because the algorithm showed it to users who were more prone to respond.”

What it means

On a platform that optimizes delivery, your A/B test is partly a test of the algorithm’s targeting. So when an ad wins, was it the better ad, or did it just happen to find better prospects? The test by itself can’t tell you.

The paper’s full title covers both halves, what these tests can’t tell you and what they can. What they can tell you is which ad will do best on that platform, delivered the way the platform delivers it, and for that the authors’ advice is to “carry on using the available A/B testing tools.” What the test can’t tell you is why the ad won, so the lesson doesn’t carry over to another channel or audience.

Where it shows up in a revenue team

• Creative tests on Meta and Google. A creative can win the test just by being shown to better prospects, and the report can’t tell you which happened.
• The algorithm chases the conversion you define. If you define it as any lead, each ad drifts toward whoever converts cheapest for it, and those are rarely the same people.
• Judge creative on what comes after. Compare the value of the customers each version brought in a few months later, and not only the conversion rate in the test.