Incident: The wrong number measured, very accurately

Module 3 · Incident file · About 35 minutes

What happened

In October 2019 Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan published "Dissecting racial bias in an algorithm used to manage the health of populations" (Science 366, 447-453, 25 October 2019, doi.org/10.1126/science.aax2342). They studied a commercial algorithm that health systems use to pick patients for "high-risk care management" programmes: extra nurses, extra appointments, closer coordination. By industry estimates, tools of this class "are applied to roughly 200 million people in the United States each year". The paper does not name the manufacturer.

Working with "a large academic hospital", the team took all primary care patients enrolled in risk-based contracts from 2013 to 2015: 6,079 who self-identified as Black and 43,539 who self-identified as White. Patients above the 97th percentile of the algorithm's score were "automatically identified for enrollment"; those above the 55th were referred to their doctor.

What it cost: "At a given risk score, Black patients are considerably sicker than White patients." At the 97th percentile, Black patients had "26.3% more chronic illnesses than Whites": 4.8 against 3.8 distinct conditions. Remedying the disparity "would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%."

The authors then contacted the manufacturer, which "independently replicated our analyses on its national dataset of 3,695,943 commercially insured patients" and confirmed the result.