Library Research Humans, algorithms and who decides
The Algorithm Worked, but It Was Aimed at the Wrong Thing
A widely used health algorithm chose patients for extra care by predicting their medical costs. Changing what it predicted would have more than doubled the share of Black patients it flagged for help.
Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, 2019
Percentage of Black patients receiving additional help
What they were looking at
Hospitals and insurers use algorithms to decide which patients get extra care, like a nurse who checks in, closer follow-up or help managing a chronic condition. Ziad Obermeyer and his colleagues audited one widely used commercial algorithm of this kind.
Their data came from primary-care patients at a large academic hospital from 2013 to 2015. That was 6,079 Black patients and 43,539 White patients. Patients above the 97th percentile of the algorithm’s risk score were enrolled in the program automatically, and those above the 55th were referred to their doctor to consider.
What they found
“At a given risk score, Black patients are considerably sicker than White patients” Obermeyer et al. (2019), abstract
So where did that come from? It came from what the algorithm was told to predict.
“The bias arises because the algorithm predicts health care costs rather than illness” Same
Less money is spent on Black patients with the same level of need, so a prediction of cost ranked them as healthier than they were. Remedying that, the authors estimate, would raise the percentage of Black patients receiving extra help from 17.7% to 46.5%. They also give a general warning.
“the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias” Same
What it means
Nothing in the algorithm was broken, and it predicted cost very well. The trouble is that cost was a convenient, measurable stand-in for need, and the harm came from the gap between the two.
This case is about people’s health and about race, and it deserves to be read on its own terms first. It also happens to be the clearest published example of a problem every team that uses an algorithm runs into. The model does exactly what it’s pointed at, so choosing the target is the most important decision in the whole system.
Where it shows up in a revenue team
- Lead volume, conversion count and cost per lead are convenient proxies for value.
- Tell an ad platform to find conversions and it’ll find conversions, very well. Whether they’re worth anything depends on what you told it a conversion is.
- Fix the target as well as the model. Define the conversion around the outcome you actually want, and check that the people it finds look like your best customers.