There’s an interesting paper in Science by Ziad Obermeyer and colleagues on racial bias in healthcare risk prediction algorithms:
The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients. Thus, despite health care cost appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise. We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.
The effect of this bias is that healthier White patients were referred for extra services in place of less-healthy Black patients (who presumably needed the extra services more).
This is a good reminder for anyone working on healthcare models to always stratify to see if the model behaves differently for traditionally marginalized groups. Not fixing these issues has serious real-world consequences on people’s health!