Library Research Measurement, experiments and ad platforms
The Drugs Improved the Number and More Patients Died
In the 1980s, heart-attack patients were given drugs that suppressed irregular heartbeats, a warning sign of sudden death. A trial found the drugs improved the number and worsened the outcome.
Debra S. Echt et al., 1991 (the CAST trial); Thomas R. Fleming and David L. DeMets, 1996
Deaths from arrhythmia in the trial
What they were looking at
After a heart attack, many patients have irregular heartbeats, and those patients are more likely to die suddenly. In the 1980s there were drugs that suppressed the irregular beats. So the logic looked airtight. Suppress the beats and you prevent the deaths, right?
The Cardiac Arrhythmia Suppression Trial, known as CAST, was set up to check that logic. It was
“designed to test the hypothesis that suppression of ventricular ectopy after a myocardial infarction reduces the incidence of sudden death” Echt et al. (1991), abstract
Patients whose irregular beats could be suppressed by the drugs were randomly assigned to a drug or a placebo.
What they found
The drugs did what they were supposed to do to the heartbeat. And the patients taking them died more often.
“The use of encainide and flecainide was discontinued because of excess mortality.” Same
Of the 1,498 patients in those two parts of the trial, 43 on the drugs died of arrhythmia, against 16 on placebo. The preliminary report in 1989 counted all deaths, and it came to 56 of 730 patients on the drugs against 22 of 725 on placebo.
Five years later the statisticians Thomas Fleming and David DeMets gathered examples from several areas of medicine and asked how often medicine gets misled this way. A surrogate end point is the number you can measure now in place of the outcome you care about. For it to be useful, it has to reliably predict the effect on that outcome. How often does that hold up?
“In practice, this requirement frequently fails.” Fleming & DeMets (1996), abstract
What it means
A surrogate is only safe to optimize if moving it moves the outcome. Here the drugs moved the surrogate the right way and the outcome the wrong way, and nobody could see it until a trial counted the outcome itself.
This is the starkest example on record, and it involves real deaths, so it’s worth being careful with. What carries over to a revenue team is narrower, but it still matters. An improved proxy doesn’t prove anything on its own.
Where it shows up in a revenue team
- MQLs, click-through rate, form fills, cost per lead and approval rate are all surrogates for revenue.
- Check each one against the outcome before you optimize to it. If the leads that pass your score don’t turn into better customers, your score is a suppressed heartbeat.
- Keep counting the outcome. Track what each month’s cohort of new customers actually returns over time, along with how the leading indicators moved.