Library Research Selection bias and paradoxes
When the Total Points the Wrong Way
In 1973 Berkeley's graduate admissions looked biased against women. Department by department, they weren't. The difference was the mix, and the same thing happens in your churn rate.
E. H. Simpson, 1951; P. J. Bickel, E. A. Hammel and J. W. O'Connell, 1975
Churn by tier, last year and this year
What they were looking at
In the fall of 1973, 8,442 men and 4,321 women applied to graduate programs at the University of California, Berkeley. About 44% of the men got in, and about 35% of the women.
Why did anyone dig into it? Eugene Hammel, an associate dean, looked into the numbers because, as the statistician Peter Bickel later recalled, “he was worried that the government would cut funding, on the grounds that Berkeley was biased.” Bickel, Hammel and J. W. O’Connell went through the data department by department. (The story that Berkeley was sued over it gets told a lot, but it appears to be unsupported.)
What they found
What did the totals show? The paper says the aggregate data
“shows a clear but misleading pattern of bias against female applicants.” Bickel, Hammel & O’Connell (1975), abstract
But when they took it department by department and combined the results properly, the picture reversed. In the paper’s words, “there is a small but statistically significant bias in favor of women.” How does that happen? Women had applied in larger numbers to the departments that admitted a smaller share of everyone who applied, men included.
“The bias in the aggregated data stems not from any pattern of discrimination on the part of admissions committees” Same
The statistician E. H. Simpson had described the same trap in 1951, in a short note on how tables of counts combine. Bickel remembered giving Hammel “an example of what I later learned was the Simpson paradox.” The name itself came later.
What it means
A total is an average of its parts, weighted by how many are in each part. So if the weights change, the total can move when none of the parts does, or move the opposite way from every one of them.
That’s what the chart above shows. Both tiers churn less this year, and the book churns more anyway. Why? This year’s growth came from the tier that always churns the most.
Where it shows up in a revenue team
- Churn. Your blended rate can get worse while every tier improves, or get better while every tier gets worse.
- Conversion rates. A channel’s rate “drops” because it’s now sending you more of a segment that always converts lower.
- Approval rates, win rates, rep rankings. A rep with a tougher territory can lose to a weaker rep on the total and still beat them in every segment.
- The fix is simple. Break the rate out by tier or segment before you act on it, and check whether the mix moved.