UMass model was most accurate at predicting COVID-19 deaths, study finds

A memorial representing each American death from COVID-19 in Washington.STEFANI REYNOLDS/NYT

A University of Massachusetts model that combines dozens of other models to provide short-term forecasts of COVID-19 deaths in the United States performed better than the individual models, according to a study.

The ensemble model, which synthesizes model forecasts collected by researchers at UMass Amherst, “provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it,” researchers said in a study published last week in the journal PNAS.

“Synthesizing multiple models removes the risk of overreliance on any single approach for accuracy or stability,” the study said. It said ensemble approaches have previously demonstrated “superior performance” in forecasting flu, Ebola, and dengue fever.

Combining multiple models contributes to “robustness and ability to overcome individual model biases. This is a really important consideration for public health agencies when using forecasts to inform policies,” lead author Estee Cramer, a doctoral candidate in the UMass School of Public Health and Health Sciences, said in a statement from the university.

The Centers for Disease Control and Prevention partnered with the laboratory of UMass professor Nicholas Reich to fund and create the COVID-19 Forecast Hub in early April 2020. Researchers collect COVID-19 predictions from around the world from academic, industry, and independent research groups. The modelers’ approaches, data sources, and assumptions vary, according to the study.

The study looked at models’ performance from April 2020 through October 2021. The study’s coauthors included nearly 300 researchers from the dozens of modeling groups who submitted predictions to the Forecast Hub.

The CDC posts to the Web the ensemble forecasts generated by the Forecast Hub for cases, hospitalizations, and deaths. The study did not look at the performance of forecasts for cases and hospitalizations, which researchers have said are harder to predict.

“It has been an incredible experience to collaborate directly with so many talented and motivated groups,” Reich, a biostatistician and the senior author of the paper, said in the university statement.

All the models, including the ensemble model, were less consistent and accurate in predicting deaths during pandemic waves, generally underpredicting deaths as trends were rising and overpredicting as they were falling, the university said.

They also became less accurate the further into the future they looked. Reich drew an analogy to weather forecasts.

“Because many of us interact with weather forecasts almost every day on our phones, we know not to trust the daily precipitation forecasts much past a two-week horizon,” he said in the statement. “But we don’t have the same intuition yet as a society about infectious disease forecasts. This work shows that the accuracy of forecasts for deaths is pretty good for the next four weeks, but at horizons of six weeks or more, the accuracy is typically substantially worse.”