Library Research Selection bias and paradoxes
The Planes That Came Back
In 1943 Abraham Wald worked out where bombers were most vulnerable using only the planes that survived. What he actually did is more useful than the story you've heard.
Abraham Wald, 1943; explained by Marc Mangel and Francisco J. Samaniego, 1984
Chance that a single hit brings the plane down, in Wald's worked example
What he was looking at
During the Second World War, Abraham Wald, a mathematician who had left Austria for the United States, worked with Columbia’s Statistical Research Group on problems for the military. One of them was aircraft vulnerability. Where on a bomber does a hit do the most damage?
The problem was that the only damage anyone could inspect was on the planes that made it home. The planes that were shot down took their bullet holes with them.
What he found
“Undoubtedly, the probability of downing a plane by a hit will depend considerably upon the part that receives the hit.” Wald, Statistical Research Group memo, reprinted in CNA CRC 432 (1980), p. 56
Wald saw that the missing planes still leave a trace. You know how many planes went out and how many came back, and how many hits the survivors took in each area. So what does it tell you when hits in one area are rare among the survivors? That area still gets hit. The planes hit there mostly didn’t come back.
He turned that into a method for estimating each area’s vulnerability from survivor data alone. In his worked example, using hypothetical numbers, one hit in the engines downed the plane 39% of the time, and one hit in the fuselage did it 5% of the time.
“for the observed data of this hypothetical example, the engine area is the most vulnerable” p. 65
The memos weren’t published at the time. The Center for Naval Analyses reprinted them in 1980, and in 1984 Marc Mangel and Francisco Samaniego explained the method for statisticians. In their words, “Abraham Wald worked on the problem of estimating the vulnerability of aircraft, using data obtained from survivors.”
You’ve heard the popular version, where Wald tells the Air Force to armor the places with no bullet holes. That part’s an embellishment. As the mathematician Bill Casselman has pointed out, the documented record is Wald’s memos and a couple of brief mentions in a colleague’s memoir. The real work is more useful anyway, because it gives you a method you can use again.
What it means
A sample of survivors is a filtered sample, and the information you need is in what it doesn’t show you. The obvious question is “what do the survivors have in common?” The one that gets you somewhere is “what would I see if the missing ones were here?”
Where it shows up in a revenue team
- Your current customers are the planes that came back. The ones who churned took their reasons with them.
- “Profile our best customers” built only from today’s book describes who survived, which isn’t the same as who you won.
- Win-loss reviews of won deals, surveys of customers who stayed, approvals with no record of the declines. All of these only look at survivors.
- Count what’s missing. Keep the leads that never converted, the applications you declined and the accounts that left, and analyze them alongside the rest.