San Francisco and New York City both reported their first COVID-19 cases during the first week of March. On March 16, San Francisco announced it was ordering residents to stay home to avoid spreading the coronavirus, and New York did the same less than a week later. But by the end of May, while San Francisco had attributed 43 deaths to COVID-19, New York City’s death count was over 20,000.
What explains the stark difference in COVID-19-related deaths between these two cities? Is the delay in the stay-at-home order responsible? What about city-specific measures taken to mitigate COVID-19 before the order? Is something else going on?
The divergent trajectories of San Francisco and New York City, while especially striking, are not unique. Worldwide, COVID-19 is having highly variable effects. Within the U.S., infections, hospitalizations and deaths have skyrocketed in nearly all major cities in the Northeast while remaining fairly low in some other metropolitan centers, such as Houston, Phoenix and San Diego.
How cities and states implemented public health interventions, such as school closures and stay-at-home orders, has varied widely. Comparing these interventions, whether they worked and for whom, can provide insights about the disease and help improve future policy decisions. But accurate comparisons aren’t simple.
The range of COVID-19 interventions implemented across the U.S. and worldwide was not random, making them difficult to compare. Among other things, population density, household sizes, public transportation use and hospital capacity may have contributed to the differences in COVID-19 deaths in San Francisco and New York City. These sorts of differences complicate analyses of the effectiveness of responses to the COVID-19 pandemic.
As a biostatistician and an epidemiologist, we use statistical methods to sort out causes and effects by controlling for the differences between communities. With COVID-19, we’ve often seen comparisons that don’t adjust for these differences. The following experiment shows why that can be a problem.
City simulations reveal a paradox
To illustrate the dangers of comparisons that fail to adjust for differences, we set up a simple computer simulation with only three hypothetical variables: city size, timing of stay-at-home orders and cumulative COVID-19 deaths by May 15.
For 300 simulated cities, we plotted COVID-19 deaths by the delay time, defined as the number of days between March 1 and the order being issued. Among cities of comparable size, delays in implementing stay-at-home orders are associated with more COVID-19 deaths – specifically, 40-63 more deaths are expected for each 10-day delay. The hypothetical policy recommendation from this analysis would be for immediate implementation of stay-at-home orders.
- ^ San Francisco announced (www.youtube.com)
- ^ New York did the same (coronavirus.health.ny.gov)
- ^ 43 deaths (data.sfgov.org)
- ^ over 20,000 (www1.nyc.gov)
- ^ Worldwide (www.nytimes.com)
- ^ Within the U.S. (coronavirus.jhu.edu)
- ^ implemented (github.com)
- ^ was not random (doi.org)
- ^ a biostatistician (theconversation.com)
- ^ an epidemiologist (theconversation.com)
- ^ computer simulation (github.com)
- ^ CC BY-ND (creativecommons.org)
- ^ paradox (en.wikipedia.org)
- ^ Correlation does not imply causation (en.wikipedia.org)
- ^ CC BY-ND (creativecommons.org)
- ^ Hispanic (www.sfgate.com)
- ^ vary widely among communities (theconversation.com)
- ^ John Snow (www.ph.ucla.edu)
- ^ causal inference methods (doi.org)
- ^ Read The Conversation’s newsletter (theconversation.com)
Authors: Laura B. Balzer, Assistant Professor of Biostatistics & Director of the UMass Causality Lab, University of Massachusetts Amherst