Measurement, experiments and ad platforms

At eBay, "brand keyword ads have no measurable short-term benefits." For non-brand keywords, frequent users whose behavior ads don't change "account for most of the advertising expenses, resulting in average returns that are negative."

Attribution gives your ads credit for buyers who were coming anyway.

Across 25 field experiments, "the median confidence interval on return on investment is over 100 percentage points wide." Informative tests "can easily require more than 10 million person-weeks," and selection bias "is a crippling concern" for observational methods.

Be suspicious of any precise ROAS number you're handed, especially when your revenue is heavy-tailed.

Across 15 experiments, "observational methods often fail to accurately recover the treatment effects" of randomized ad experiments, usually by overstating them.

A model built on platform data can't stand in for a holdout.

In a retailer's retargeting campaign, "the ads lifted website visits by 17.2% and purchases by 10.5%." Ghost ads measure lift just as precisely "while spending at least an order of magnitude less."

It's a cheaper way to test retargeting, the channel most distorted by selection.

Platforms show each ad version to "distinct and undetectably optimized mixes of users." That mixes up the effect of the ad with the effect of the targeting, and can distort "the magnitude, and even the sign" of the result.

The platform picks who sees each ad, so a creative can win just by reaching better prospects. The targeting decides your test before the ad does.

"both the advertiser's budget and the content of the ad each significantly contribute to the skew of Facebook's ad delivery," even "despite neutral targeting parameters."

Whatever your targeting settings say, the platform decides who sees your ad.

Fewer women saw a gender-neutral ad "because younger women are a prized demographic and are more expensive to show ads to." The skew came from auction prices, not targeting choices.

An algorithm that minimizes your cost drifts toward whoever is cheapest to reach, whatever they're worth.

The overall evaluation criterion (OEC) "must be measurable in the short term (the duration of an experiment) yet believed to causally drive long-term strategic objectives."

Whatever metric you test on ends up deciding which customers you select.

At Google, "the short-term effect is not always predictive of the long-term effect." Users learned to ignore low-quality ads, and the work led to "a 50% reduction of the ad load on Google's mobile search interface."

Chasing short-term revenue wins can teach your customers to tune you out.

Combining several short-term outcomes into one index, the first six quarters of a job-training experiment recovered nine-year effects "with a 35% reduction in standard errors."

It's the rigorous version of "signals," early indicators validated against long-term value.

A valid surrogate must "reliably predict the overall effect on the clinical outcome. In practice, this requirement frequently fails."

Your MQLs, CTR and form fills are surrogates. Validate them against revenue before you optimize to them.

The drugs suppressed irregular heartbeats (the surrogate) but were stopped "because of excess mortality": 43 arrhythmic deaths on the drugs against 16 on placebo.

It's the starkest case on record of a better number and a worse outcome.

Teams launch only the experiments that won, so summing their measured wins overstates the total impact. The authors estimate the bias and correct for it in Airbnb's experiment reporting.

The cumulative lift your testing program claims is inflated.

Firms that tried to contact leads within an hour were "nearly seven times as likely to qualify the lead" as those that waited an hour longer, and "more than 60 times as likely" as those that waited 24 hours or more.

Speed matters. When you fix bad routing, route on value with a short time-out so speed doesn't get lost.