What Happened When eBay Turned Off Its Search Ads

What they were looking at

eBay spent heavily on search ads, and like most advertisers it judged them the usual way. Track who clicked an ad, then see who bought. Three economists working with eBay’s own data, Thomas Blake, Chris Nosko and Steven Tadelis, asked a simpler question. What would happen if the ads just weren’t there?

They ran two experiments:

• Brand keywords. In March 2012 eBay stopped bidding on searches for “eBay” on Yahoo! and MSN, and kept bidding on Google for comparison.
• Everything else. Then it switched off its non-brand Google search ads for 60 days in about 30% of US markets (68 test areas), and compared them with the rest of the country.

What they found

Start with the brand ads.

“brand-keyword ads have no measurable short-term benefits.” Blake, Nosko & Tadelis, NBER working paper, abstract

People searching for “eBay” were going to eBay anyway. With the ad gone, they just clicked the ordinary search result instead.

The non-brand ads did move some people, mostly new and infrequent buyers. But that’s not where most of the money went:

“frequent users whose purchasing behavior is not influenced by ads account for most of the advertising expenses” Same

“returns from paid search are a fraction of conventional non-experimental estimates.” Same

In the working paper’s figures, the experiment put the return on the non-brand ads at about −63%. The usual analysis, the one that credits the ad for everyone who clicked and then bought, put it above 4,100%.

What it means

Attribution answers the question “who clicked an ad before they bought?” The experiment asks a different one, “who bought because of the ad?” For loyal customers who were coming anyway, the answer is almost nobody, and that’s where most of the budget went.

The result has limits. eBay is one of the best-known names online, and the authors note it may not hold for smaller brands. A 2018 study by Andrey Simonov, Chris Nosko and Justin Rao found brand ads added 1% to 4% more clicks when no competitor was bidding, more for smaller brands, and that a brand that stopped bidding on its own name while competitors kept going lost 18% to 42% of its clicks to them.

Where it shows up in a revenue team

• Brand search and last-click reports credit ads for customers who were on their way already. Retargeting can too, though a 2017 ghost-ads test of a retailer’s retargeting measured a real 10.5% lift in purchases, so test it before you cut it.
• “Our best-performing campaign” is the one sitting closest to people who’d have bought regardless.
• Run a holdout. Switch the spend off in some markets, or for a random slice of the audience, and compare. That’s your baseline, what happens when you take your hands off the wheel, and it’s the only way to see what the ads add.