Earlier this month, following the U.S. Federal Food and Drug Administration’s emergency approval, the federal government began distributing doses of Pfizer’s COVID-19 vaccine to health systems throughout the country. With Moderna’s vaccine to follow, the Trump administration aims to deliver the first shots to 20 million people by year’s end.
But shipments must be prioritized in a country of over 300 million people. This logistical challenge has fallen on algorithms designed to account for a range of factors in identifying which populations are most vulnerable. Problematically, however, a lack of transparency plagues their decision-making processes. And given the body of research showing algorithms can encode biases against certain demographic groups, particularly minorities and low-income earners, experts believe this is cause for concern.
Tiberius
In an effort to achieve its goal of inoculating 100 million Americans by Q1 2021, the U.S. Department of Health and Human Services partnered with data-mining firm Palantir to develop a software platform called Tiberius. Tiberius, which covers 50 states, eight territories, the Veterans Health Administration, the Bureau of Prisons, the Indian Health Service, and the departments of Defense and State, allows states and federal agencies to see health care providers enrolled in VTrckS, the U.S. Centers for Disease Control and Prevention’s (CDC) vaccine tracking system. They can also use Tiberius to make decisions about where they send the vaccine within their jurisdictions.
Each Thursday, vaccine producers inform Operation Warp Speed, the U.S. public-private COVID-19 treatment development partnership, how much vaccine is available for the upcoming week. On Friday, Tiberius runs an algorithm that draws on variables including adult population and vaccine uptake and allocates the number of doses available to every state. On Saturday, every state finalizes their orders, and deliveries arrive on Monday.
Operation Warp Speed’s chief of plans Col. Deacon Maddox has described Tiberius’ algorithm as “simple,” “equitable,” and “fair.” According to Maddox, it takes as input “risk-adjusted” manufacturing estimates, a small portion from which is subtracted to provide a “safety stock” and the remainder of which is divided by population estimates from the U.S. Census Bureau’s 2018 American Community Survey and the CIA’s World Factbook.
While it’s impossible to know exactly which variables the Tiberius algorithm considers in its decisions — that information isn’t public — its reliance on U.S. census data is cause for worry. Because of time constraints, a shortage of manpower, and other confounders, the Census Bureau regularly undercounts populations in certain regions of the country. For example, studies have shown the undercount for Black men, particularly those aged 30 to 49, is much higher than the net undercount rate for the total Black population or total male population.
The impact could be devastating. Black Americans are infected with COVID-19 at nearly three times the rate of white Americans and are more than twice as likely to die from the virus, according to data from Johns Hopkins University. That’s partly because Black people are more likely to have preexisting conditions that predispose them to infection, less likely to have health insurance, and more likely to work in jobs that don’t accommodate remote work.
“There’s this question of underreporting and overreporting. [It] doesn’t just impact assessments of how many doses are needed in a particular area, but also the priority of those doses,” Os Keyes, an AI researcher at the University of Washington’s Department of Human Centered Design and Engineering, told VentureBeat via email. “If you are not including vulnerable populations as often, you are undercounting population — which means you have a false sense of geographic density. Moreover, your sense of geographic density is missing people who, as a result of their treatment and lives, are more likely to live in dense housing, low-quality housing, places where social distancing is particularly difficult and preexisting health inequalities that exacerbate vulnerability to COVID are particularly high.”
State algorithms
Beyond Tiberius, some states including Ohio, Wisconsin, South Carolina, New Hampshire, Delaware, Tennessee, and Arizona are using algorithms developed in-house to determine how vaccines should be distributed locally.
According to a draft document outlining Ohio’s distribution plan, the state’s algorithm, which was created by the Ohio Department of Health (ODH), takes into account the number of target population groups in a county, the current case count, the level of natural immunity that might exist, labels like “social vulnerability” and “health equity,” and the amount and type of storage providers have on hand. The population estimates come from census information in addition to U.S. Bureau of Labor Statistics, nursing home, and medical claims data, while factors like social vulnerability and health equity are determined using the CDC’s Social Vulnerability Index, which uses 15 U.S. census metrics to identify disadvantaged communities that might need additional support during disasters.
The document also reveals that ODH considered geographies and populations with vulnerabilities or structural disadvantages, including low access to health care or those disproportionately affected by COVID-19, as well as locations where social distancing may be most challenging due to population density. The department also looked at where historical flu vaccination rates were lower compared with other regions. And to identify high-risk health care workers, the ODH mailed a survey to local health department grant recipients in 113 jurisdictions requiring that they identify their critical workforce.
Above: New Hampshire’s proposed vaccine distribution algorithm.
The machinations of South Carolina’s algorithm are less immediately clear. According to a state-issued document, it accounts for the number of recipients a site can vaccinate, a site’s vaccine storage and handling capacity, a site’s geographic location, and allocations of vaccine received and timing. But it’s unclear whether the algorithm factors in population data and from where this population data might be sourced. Moreover, South Carolina’s document makes only passing mention of efforts to ensure disadvantaged groups are fairly represented.
In its draft distribution plan, Tennessee says it’s distributing the Pfizer vaccine across “qualifying facilities” by hand but plans to use an algorithm to allocate the Moderna vaccine. The document doesn’t describe this algorithm in detail, however, or identify which department will be responsible for implementing it and the sources of data that might influence its predictions. Arizona and Wisconsin have been similarly vague about where, when, and how their algorithms might be used; Arizona Department of Health Services officials have revealed only that their algorithm, which was provided by the CDC, prioritizes health care workers with the most exposure to people with COVID-19 as well as those with underlying conditions.
Perpetuating inequality
It’s not just states that are using — or pledging to use — algorithms to determine who receives the vaccine first (and who doesn’t). At George Washington University Hospital (GWUH), groups are selected by a system that takes into account age, underlying medical conditions, prevention of transmission, and the hospital’s ability to continue its own operations. At press time, the GWUH hadn’t responded to VentureBeat’s request for more information.
Keyes raises the concern that it’s unclear, in many cases, how organizations and agencies using algorithms plan to supplement missing data. He notes that the CDC’s Social Vulnerability Index only began canvassing Puerto Rico in 2014 — the only U.S. territory it covers — and that the American Community Survey includes only a handful of territories and tribal nations.
Problems have already begun to emerge. Yesterday, ProPublica reported that Stanford Medicine residents who work in close contact with COVID-19 patients were left out of the initial wave for the Pfizer vaccine. An algorithm chose who would be the first 5,000 in line; the residents were told they were at a disadvantage because they lacked an assigned “location” to plug into the calculation and because they’re young.
“We are writing to acknowledge the significant concerns expressed by our community regarding the development and execution of our vaccine distribution plan,” leaders of Stanford Health Care and Stanford School of Medicine wrote in an email to staff apologizing for the debacle, NPR reported on Friday evening. “We take complete responsibility and profusely apologize to all of you. We fully recognize we should have acted more swiftly to address the errors that resulted in an outcome we did not anticipate,” they wrote.
“When I hear you’re covering 50 states and eight territories with data determined, in part, from these sources, I want to know what on earth you’re doing when some of those sources just aren’t available,” Keyes said. “Working it out on the back of an envelope? Giving up entirely? The answer might be a good one, and the approach might be the best one they can take at the time, but it’s worrisome that the answer isn’t, as far as I can tell, known.”
As vaccine distribution ramps up into next year, it’s likely that federal and state agencies and health care systems will respond to distribution kinks by refining their algorithms. Already, officials in more than a dozen states have complained they’re receiving fewer doses than promised, and Pfizer claims the federal government hasn’t issued shipping instructions for millions of doses currently sitting in warehouses. Still, there’s cause for worry in these early stages. A lack of transparency, combined with unbalanced sources of data and methodologies, threaten to cement health care inequalities in communities throughout the U.S.
