To install one of these extensions in Brackets, choose File > Extension Manager and click on the Available tab.
Extension Description Productivity / Project Files Search1.5.0 on 2023-01-16 More Info... Search within your project files. Folder and file names are considered. Separate search chunks by space. Visit homepage for full info. Keywords: delete line start line end
Epidemic forecasting has a dubious track-record, and its failures became more prominent with COVID-19. Poor data input, wrong modeling assumptions, high sensitivity of estimates, lack of incorporation of epidemiological features, poor past evidence on effects of available interventions, lack of transparency, errors, lack of determinacy, consideration of only one or a few dimensions of the problem at hand, lack of expertise in crucial disciplines, groupthink and bandwagon effects, and selective reporting are some of the causes of these failures. Nevertheless, epidemic forecasting is unlikely to be abandoned. Some [but not all] of these problems can be fixed. Careful modeling of predictive distributions rather than focusing on point estimates, considering multiple dimensions of impact, and continuously reappraising models based on their validated performance may help. If extreme values are considered, extremes should be considered for the consequences of multiple dimensions of impact so as to continuously calibrate predictive insights and decision-making. When major decisions [e.g. draconian lockdowns] are based on forecasts, the harms [in terms of health, economy, and society at large] and the asymmetry of risks need to be approached in a holistic fashion, considering the totality of the evidence.
Keywords: Forecasting, COVID-19, Mortality, Hospital bed utilization, Bayesian models, SIR models, Bias, Validation
1. Initial position
COVID-19 is a major acute crisis with unpredictable consequences. Many scientists have struggled to make forecasts about its impact []. However, despite involving many excellent modelers, best intentions, and highly sophisticated tools, forecasting efforts have largely failed.
Early on, experienced modelers drew parallels between COVID-19 and the Spanish flu [//www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/report-9-impact-of-npis-on-covid-19/. [Accessed 2 June 2020]] that caused >50 million deaths with mean age of death being 28. We all lament the current loss of life. However, as of June 18, the total fatalities are ∼450,000 with median age ∼80 and typically multiple comorbidities.
Brilliant scientists expected 100,000,000 cases accruing within 4 weeks in the USA []. Predictions for hospital and ICU bed requirements were also entirely misinforming. Public leaders trusted models [sometimes even black boxes without disclosed methodology] inferring massively overwhelmed health care capacity [Table 1] []. However, very few hospitals were eventually stressed and only for a couple of weeks. Most hospitals maintained largely empty wards, expecting tsunamis that never came. The general population was locked and placed in horror-alert to save health systems from collapsing. Tragically, many health systems faced major adverse consequences, not by COVID-19 cases overload, but for very different reasons. Patients with heart attacks avoided hospitals for care [], important treatments [e.g. for cancer] were unjustifiably delayed [] and mental health suffered []. With damaged operations, many hospitals started losing personnel, reducing their capacity to face future crises [e.g. a second wave]. With massive new unemployment, more people may lose health insurance. The prospects of starvation and of lack of control of other infectious diseases [such as tuberculosis, malaria, and childhood communicable diseases where vaccination is hindered by COVID-19 measures] are dire [, ].
Table 1
Some predictions about hospital bed needs and their rebuttal by reality: examples from news coverage of some influential forecasts.
StatePrediction madeWhat happenedNew York [//www.nytimes.com/2020/04/10/ nyregion/new-york-coronavirus-hospitals.html and ]“Sophisticated scientists, Mr. Cuomo said, had studied the coming coronavirus outbreak and their projections were alarming. Infections were doubling nearly every three days and the state would soon require an unthinkable expansion of its health care system. To stave off a catastrophe, New York might need up to 140,000 hospital beds and as many as 40,000 intensive care units with ventilators.” 4/10/2020“But the number of intensive care beds being used declined for the first time in the crisis, to 4,908, according to daily figures released on Friday. And the total number hospitalized with the virus, 18,569, was far lower than the darkest expectations.” 4/10/2020
“Here’s my projection model. Here’s my projection model. They were all wrong. They were all wrong.” Governor Andrew Cuomo 5/25/2020
Tennessee [//www.nashvillepost.com/business/health-care/article/21127025/covid19-update-hospitalization-projections-drop and //www.tennessean.com/story/money/industries/health-care/2020/06/04/tennessee-hospitals-expected-lose-3-5-billion-end-june/3139003001/]“Last Friday, the model suggested Tennessee would see the peak of the pandemic on about April 19 and would need an estimated 15,500 inpatient beds, 2,500 ICU beds and nearly 2,000 ventilators to keep COVID-19 patients alive.”“Now, it is projecting the peak to come four days earlier and that the state will need 1,232 inpatients beds, 245 ICU beds and 208 ventilators. Those numbers are all well below the state’s current health care capacity.”
“Hospitals across the state will lose an estimated $3.5 billion in revenue by the end of June because of limitations on surgeries and a dramatic decrease in patients during the coronavirus outbreak, according to new estimates from the Tennessee Hospital Association.” 6/4/2020
California [ //www.sacbee.com/news/california/article241621891.html and //medicalxpress.com/news/2020-04-opinion-hospitals-beds-non-covid-patients.html]“In California alone, at least 1.2 million people over the age of 18 are projected to need hospitalization from the disease, according to an analysis published March 17 by the Harvard Global Health Institute and the Harvard T.H. Chan School of Public Health… California needs 50,000 additional hospital beds to meet the incoming surge of coronavirus patients, Gov. Gavin Newsom said last week.”“In our home state of California, for example, COVID-19 patients occupy fewer than two in 10 ICU beds, and the growth in COVID-19-related utilization, thankfully, seems to be flattening out. California’s picture is even sunnier when it comes to general hospital beds. Well under five percent are occupied by COVID-19 patients.”
Modeling resurgence after reopening also failed [Table 2]. For example, a Massachusetts General Hospital model [//www.massgeneral.org/news/coronavirus/COVID-19-simulator. [Accessed 2 June 2020]] predicted over 23,000 deaths within a month of Georgia reopening – the actual deaths were 896.
Table 2
Forecasting what will happen after reopening.
PREDICTION FOR REOPENINGWHAT ACTUALLY HAPPENED“Results indicate that lifting restrictions too soon can result in a second wave of infections and deaths. Georgia is planning to open some businesses on April 27th. The tool shows that COVID-19 is not yet contained in Georgia and even lifting restrictions gradually over the next month can result in over 23,000 deaths.”
Massachusetts General Hospital News, April 24, 2020
[//www.massgeneral.org/news/coronavirus/COVID-19-simulator]
Number of deaths over one month: 896 instead of the predicted 23,000
“administration is privately projecting a steady rise in the number of coronavirus cases and deaths over the next several weeks. The daily death toll will reach about 3,000 on June 1, according to an internal document obtained by The New York Times, a 70 percent increase from the current number of about 1,750.
The projections, based on government modeling pulled together by the Federal Emergency Management Agency, forecast about 200,000 new cases each day by the end of the month, up from about 25,000 cases a day currently.”
New York Times, May 4, 2020 [ //www.nytimes.com/2020/05/04/us/coronavirus-live-updates.html]
Number of daily deaths on June 1: 731 instead of the predicted 3,000, i.e. a 60% decrease instead of 70% increase
Number of daily new cases on May 31: 20,724 instead of the predicted 125,000, i.e. a 15% decrease instead of 700% increase
“According to the Penn Wharton Budget Model [PWBM], reopening states will result in an additional 233,000 deaths from the virus — even if states don’t reopen at all and with social distancing rules in place. This means that if the states were to reopen, 350,000 people in total would die from coronavirus by the end of June, the study found.”
Yahoo, May 3, 2020 [ //www.yahoo.com/now/reopening-states-will-cause-233000-more-people-to-die-from-coronavirus-according-to-wharton-model-120049573.html]
Based on JHU dashboard death count, number of additional deaths as of June 30 was 5,700 instead of 233,000, i.e. total deaths was 122,700 instead of 350,000. It is unclear also whether any of the 5,700 deaths were due to reopening rather than error in the original model calibration of the number of deaths without reopening.
“Dr. Ashish Jha, the director of the Harvard Global Health Institute, told CNN’s Wolf Blitzer that the current data shows that somewhere between 800 to 1000 Americans are dying from the virus daily, and even if that does not increase, the US is poised to cross 200,000 deaths sometime in September.
“I think that is catastrophic. I think that is not something we have to be fated to live with,” Jha told CNN. “We can change the course. We can change course today.”
“We’re really the only major country in the world that opened back up without really getting our cases as down low as we really needed to,” Jha told CNN.”
Business Insider, June 10, 2020 [//www.businessinsider.com/harvard-expert-predicts-coronavirus-deaths-in-us-by-september-2020-6]
Within less than 4 weeks of this quote, the number of daily deaths was much less than the 800–1000 quote [516 daily average for the week ending July 4]. Then it increased again to over 1000 daily average in the first three weeks in August and then it decreased again to 710 daily average by the last week of September. Predictions are precarious with such volatile behavior of the epidemic wave.
Table 3 lists some main reasons underlying this forecasting failure. Unsurprisingly, models failed when they used more speculation and theoretical assumptions and tried to predict long-term outcomes; for example, using early SIR-based models to predict what would happen in the entire season. However, even forecasting built directly on data alone fared badly [, ], failing not only in ICU bed predictions [Fig. 1] but also in next day death predictions when issues of long-term chaotic behavior do not come into play [Fig. 2, Fig. 3]. Even for short-term forecasting when the epidemic wave waned, models presented confusingly diverse predictions with huge uncertainty [Fig. 4].
Table 3
Potential reasons for the failure of COVID-19 forecasting along with examples and extent of potential amendments.
ReasonsExamplesHow to fix: extent of potential amendmentsPoor data input on key features of the pandemic that go into theory-based forecasting [e.g. SIR models]Early data providing estimates for case fatality rate, infection fatality rate, basic reproductive number, and other key numbers that are essential in modeling were inflated.May be unavoidable early in the course of the pandemic when limited data are available; should be possible to correct when additional evidence accrues about true spread of the infection, proportion of asymptomatic and non-detected cases, and risk-stratification. Investment should be made in the collection, cleaning, and curation of data.
Poor data input for data-based forecasting [e.g. time series]Lack of consensus as to what is the ‘ground truth” even for seemingly hard-core data such as the daily the number of deaths. They may vary because of reporting delays, changing definitions, data errors, etc. Different models were trained on different and possibly highly inconsistent versions of the data.As above: investment should be made in the collection, cleaning, and curation of data.
Wrong assumptions in the modelingMany models assume homogeneity, i.e. all people having equal chances of mixing with each other and infecting each other. This is an untenable assumption and, in reality, tremendous heterogeneity of exposures and mixing is likely to be the norm. Unless this heterogeneity is recognized, estimates of the proportion of people eventually infected before reaching herd immunity can be markedly inflatedNeed to build probabilistic models that allow for more realistic assumptions; quantify uncertainty and continuously re-adjust models based on accruing evidence
High sensitivity of estimatesFor models that use exponentiated variables, small errors may result in major deviations from realityInherently impossible to fix; can only acknowledge that uncertainty in calculations may be much larger than it seems
Lack of incorporation of epidemiological featuresAlmost all COVID-19 mortality models focused on number of deaths, without considering age structure and comorbidities. This can give very misleading inferences about the burden of disease in terms of quality-adjusted life-years lost, which is far more important than simple death count. For example, the Spanish flu killed young people with average age of 28 and its burden in terms of number of quality-adjusted person-years lost was about 1000-fold higher than the COVID-19 [at least as of June 8, 2020].Incorporate best epidemiological estimates of age structure and comorbidities in the modeling; focus on quality-adjusted life-years rather than deaths
Poor past evidence on effects of available interventionsThe core evidence to support “flatten-the-curve” efforts was based on observational data from the 1918 Spanish flu pandemic on 43 US cites. These data are >100-years old, of questionable quality, unadjusted for confounders, based on ecological reasoning, and pertaining to an entirely different [influenza] pathogen that had ∼100-fold higher infection fatality rate than SARS-CoV-2. Even thus, the impact on reduction of total deaths was of borderline significance and very small [10%–20% relative risk reduction]; conversely, many models have assumed a 25-fold reduction in deaths [e.g. from 510,000 deaths to 20,000 deaths in the Imperial College model] with adopted measuresWhile some interventions in the broader package of lockdown measures are likely to have beneficial effects, assuming huge benefits is incongruent with past [weak] evidence and should be avoided. Large benefits may be feasible from precise, focused measures [e.g. early, intensive testing with thorough contact tracing for the early detected cases, so as not to allow the epidemic wave to escalate [e.g. Taiwan or Singapore]; or draconian hygiene measures and thorough testing in nursing homes] rather than from blind lockdown of whole populations.
Lack of transparencyThe methods of many models used by policy makers were not disclosed; most models were never formally peer-reviewed, and the vast majority have not appeared in the peer-reviewed literature even many months after they shaped major policy actionsWhile formal peer-review and publication may unavoidably take more time, full transparency about the methods and sharing of the code and data that inform these models is indispensable. Even with peer-review, many papers may still be glaringly wrong, even in the best journals.
ErrorsComplex code can be error-prone, and errors can happen even by experienced modelers; using old-fashioned software or languages can make things worse; lack of sharing code and data [or sharing them late] does not allow detecting and correcting errorsPromote data and code sharing; use up-to-date and well-vetted tools and processes that minimize the potential for error through auditing loops in the software and code
Lack of determinacyMany models are stochastic and need to have a large number of iterations run, perhaps also with appropriate burn-in periods; superficial use may lead to different estimatesPromote data and code sharing to allow checking the use of stochastic processes and their stability
Looking at only one or a few dimensions of the problem at handAlmost all models that had a prominent role in decision-making focused on COVID-19 outcomes, often just a single outcome or a few outcomes [e.g. deaths or hospital needs]. Models prime for decision-making need to take into account the impact on multiple fronts [e.g. other aspects of health care, other diseases, dimensions of the economy, etc.]Interdisciplinarity is desperately needed; as it is unlikely that single scientists or even teams can cover all this space, it is important for modelers from diverse ways of life to sit at the same table. Major pandemics happen rarely, and what is needed are models which combine information from a variety of sources. Information from data, from experts in the field, and from past pandemics, need to combined in a logically consistent fashion if we wish to get any sensible predictions.
Lack of expertise in crucial disciplinesThe credentials of modelers are sometimes undisclosed; when they have been disclosed, these teams are led by scientists who may have strengths in some quantitative fields, but these fields may be remote from infectious diseases and clinical epidemiology; modelers may operate in subject matter vacuumMake sure that the modelers’ team is diversified and solidly grounded in terms of subject matter expertise
Groupthink and bandwagon effectsModels can be tuned to get desirable results and predictions; e.g. by changing the input of what are deemed to be plausible values for key variables. This is especially true for models that depend on theory and speculation, but even data-driven forecasting can do the same, depending on how the modeling is performed. In the presence of strong groupthink and bandwagon effects, modelers may consciously fit their predictions to what is the dominant thinking and expectations – or they may be forced to do so.Maintain an open-minded approach; unfortunately, models are very difficult, if not impossible, to pre-register, so subjectivity is largely unavoidable and should be taken into account in deciding how much forecasting predictions can be trusted
Selective reportingForecasts may be more likely to be published or disseminated if they are more extremeVery difficult to diminish, especially in charged environments; needs to be taken into account in appraising the credibility of extreme forecasts
![An external file that holds a picture, illustration, etc. Object name is gr1_lrg.jpg][////i0.wp.com/www.ncbi.nlm.nih.gov/pmc/articles/PMC7447267/bin/gr1_lrg.jpg]
Predictions for ICU beds made by the IHME models on March 31 for three states: California, New Jersey, and New York. For New York, the model initially over predicted enormously, and then it under predicted. For New Jersey, a neighboring state, the model started well but then it underpredicts, while for California it predicted a peak which never eventuated.
![An external file that holds a picture, illustration, etc. Object name is gr2_lrg.jpg][////i0.wp.com/www.ncbi.nlm.nih.gov/pmc/articles/PMC7447267/bin/gr2_lrg.jpg]
Performance of four data-driven models, IHME, YYG, UT, and LANL, used to predict COVID-19 death counts by state in the USA for the following day. That is, these were predictions made only 24 h in advance of the day in question. The Figure shows the percentage of times that a particular model’s prediction was within 10% of the ground truth by state. All models failed in terms of accuracy; for the majority of states, this figure was less than 20%.
![An external file that holds a picture, illustration, etc. Object name is gr3_lrg.jpg][////i0.wp.com/www.ncbi.nlm.nih.gov/pmc/articles/PMC7447267/bin/gr3_lrg.jpg]
Performance of the same models examined in Fig. 2 in terms of their uncertainty quantification. If a model assessment of uncertainty is accurate, then we would expect 95% of the ground truth values to fall within the 95% prediction interval. Only one of the 4 models [the UT model] approached this level of accuracy.
![An external file that holds a picture, illustration, etc. Object name is gr4_lrg.jpg][////i0.wp.com/www.ncbi.nlm.nih.gov/pmc/articles/PMC7447267/bin/gr4_lrg.jpg]
Snapshot from //reichlab.io/covid19-forecast-hub/ [a very useful site that collates information and predictions from multiple forecasting models] as of 11.14 AM PT on June 3, 2020. Predictions for number of US deaths during week 27 [only ∼3 weeks downstream] with these 8 models ranged from 2419 to 11190, which is a 4.5-fold difference, and the spectrum of 95% confidence intervals ranged from fewer than 100 deaths to over 16,000 deaths, which is almost a 200-fold difference.
Failure in epidemic forecasting is an old problem. In fact, it is surprising that epidemic forecasting has retained much credibility among decision-makers, given its dubious track record. Modeling for swine flu predicted 3,100–65,000 deaths in the UK [//www.theguardian.com/uk/2009/jul/16/swine-flu-cases-rise-britain. [Accessed 2 June 2020]]. Eventually, 457 deaths occurred []. Models on foot-and-mouth disease by top scientists in top journals [, ] were subsequently questioned [] by other scientists challenging why up to 10 million animals had to be slaughtered. Predictions for bovine spongiform encephalopathy expected up to 150,000 deaths in the UK []. However, the lower bound predicted as low as 50 deaths [], which is a figure close to eventual fatalities. Predictions may work in “ideal”, isolated communities with homogeneous populations, not the complex current global world.
Despite these obvious failures, epidemic forecasting continued to thrive, perhaps because vastly erroneous predictions typically lacked serious consequences. In fact, erroneous predictions may have even been useful. A wrong, doomsday prediction may incentivize people towards better personal hygiene. Problems emerge when public leaders take [wrong] predictions too seriously, considering them crystal balls without understanding their uncertainty and the assumptions made. Slaughtering millions of animals may aggravate animal business stakeholders – but most citizens are not directly affected. However, with COVID-19, espoused wrong predictions can devastate billions of people in terms of the economy, health, and societal turmoil at-large.
Let us be clear: even if millions of deaths did not happen this season, they may happen with the next wave, next season, or some new virus in the future. A doomsday forecast may come in handy to protect civilization when and if calamity hits. However, even then, we have little evidence that aggressive measures focusing only on a few dimensions of impact actually reduce death toll and do more good than harm. We need models which incorporate multicriteria objective functions. Isolating infectious impact, from all other health, economic, and social impacts is dangerously narrow-minded. More importantly, with epidemics becoming easier to detect, opportunities for declaring global emergencies will escalate. Erroneous models can become powerful, recurrent disruptors of life on this planet. Civilization is threatened by epidemic incidentalomas.
Cirillo and Taleb thoughtfully argue [] that when it comes to contagious risk, we should take doomsday predictions seriously: major epidemics follow a fat-tail pattern and extreme value theory becomes relevant. Examining 72 major epidemics recorded through history, they demonstrate a fat-tailed mortality impact. However, they analyze only the 72 most-noticed outbreaks, which is a sample with astounding selection bias. For example, according to their dataset, the first epidemic originating from sub-Saharan Africa did not occur until 1920 AD, namely HIV/AIDS. The most famous outbreaks in human history are preferentially selected from the extreme tail of the distribution of all outbreaks. Tens of millions of outbreaks with a couple deaths must have happened throughout time. Around hundreds of thousands might have claimed dozens of fatalities. Thousands of outbreaks might have exceeded 1000 fatalities. Most eluded the historical record. The four garden variety coronaviruses may be causing such outbreaks every year [, ]. One of them, OC43 seems to have been introduced in humans as recently as 1890, probably causing a “bad influenza year” with over a million deaths []. Based on what we know now, SARS-CoV-2 may be closer to OC43 than SARS-CoV-1. This does not mean it is not serious: its initial human introduction can be highly lethal, unless we protect those at risk.
A heavy tail distribution ceases to be as heavy as Taleb imagines when the middle of the distribution becomes much larger. One may also argue that pandemics, as opposed to epidemics without worldwide distribution, are more likely to be heavy-tailed. However, the vast majority of the 72 contagious events listed by Taleb were not pandemics, but localized epidemics with circumscribed geographic activity. Overall, when a new epidemic is detected, it is even difficult to pinpoint which distribution of which known events it should be mapped against.
Blindly acting based on extreme value theory alone would be sensible if we lived in the times of the Antonine plague or even in 1890, with no science to identify the pathogen, elucidate its true prevalence, estimate accurately its lethality, and carry out good epidemiology to identify which people and settings are at risk. Until we accrue this information, immediate better-safe-than-sorry responses are legitimate, trusting extreme forecasts as possible [not necessarily likely] scenarios. However, caveats of these forecasts should not be ignored [, ] and new evidence on the ground truth needs continuous reassessment. Upon acquiring solid evidence about the epidemiological features of new outbreaks, implausible, exaggerated forecasts [] should be abandoned. Otherwise, they may cause more harm than the virus itself.
2. Further thoughts – analogies, decisions of action, and maxima
The insightful recent essay of offers additional opportunities for fruitful discussion.
2.1. Point estimate predictions and technical points
ruminates on the point of making point predictions. Serious modelers [whether frequentist or Bayesian] would never rely on point estimates to summarize skewed distributions. Even an early popular presentation [] from 1954 has a figure [see page 33] with striking resemblance to Taleb’s Fig. 1 []. In a Bayesian framework, we rely on the full posterior predictive distribution, not single points []. Moreover, Taleb’s choice of a three-parameter Pareto distribution is peculiar. It is unclear whether this model provides a measurably better fit to his [hopelessly biased] pandemic data [] than, say, a two parameter Gamma distribution fitted to log counts. Regardless, either skewed distribution would then have to be modified to allow for the use of all available sources of information in a logically consistent fully probabilistic model, such as via a Bayesian hierarchical model [which can certainly be formulated to accommodate fat tails if needed]. In this regard, we note that upon examining the NY daily death count data studied in , these data are found to be characterized as stochastic rather than chaotic []. Taleb seems to fit an unorthodox model, and then abandons all effort to predict anything. He simply assumes doomsday has come, much like a panic-driven Roman would have done in the Antonine plague, lacking statistical, biological, and epidemiological insights.
2.2. Should we wait for the best evidence before acting?
caricatures the position of a hotly debated mid-March op-ed by one of us, suggesting that it “made statements to the effect that one should wait for “more evidence” before acting with respect to the pandemic”, which is an obvious distortion of the op-ed. Anyone who reads the op-ed unbiasedly realizes that it says exactly the opposite. It starts with the clear, unquestionable premise that the pandemic is taking hold and is a serious threat. Immediate lockdown certainly makes sense when an estimated 50 million deaths are possible. This is stated emphatically on multiple occasions these days in interviews in multiple languages -for examples see , , . Certainly, adverse consequences of short-term lockdown cannot match 50 million lives. However, better data can help recalibrate estimates, re-assessing downstream the relative balance of benefits and harms of longer-term prolongation of lockdown. That re-appraised balance changed markedly over time [].
Another gross distortion propagated in social media is that the op-ed [] had supposedly predicted that only 10,000 deaths will happen in the USA as a result of the pandemic. The key message of the op-ed was that we lack reliable data, that is, we do not know. The self-contradicting misinterpretation as “we don’t know, but actually we do know that 10,000 deaths will happen” is impossible. The op-ed discussed two extreme scenarios to highlight the tremendous uncertainty absent reliable data: an overtly optimistic scenario of only 10,000 deaths in the US and an overtly pessimistic scenario of 40,000,000 deaths. We needed reliable data, quickly, to narrow this vast uncertainty. We did get data and did narrow uncertainty. Science did work eventually, even if forecasts, including those made by one of us [confessed and discussed in ], failed.
2.3. Improper and proper analogies of benefit-risk
offers several analogies to assert that all precautionary actions are justified in pandemics, deriding “waiting for the accident before putting the seat belt on, or evidence of fire before buying insurance” []. The analogies assume that the cost of precautionary actions are small in comparison to the cost of the pandemic, and that the consequences of the action have little impact on it. However, precautionary actions can backfire severely when they are misinformed. In March, modelers were forecasting collapsed health systems; for example 140,000 beds would be needed in New York, when only a small fraction were available. Precautionary actions damaged the health system, increased COVID-19 deaths [], and exacerbated other health problems [Table 4].
Table 4
Adverse consequences of precautionary actions, expecting excess of hospitalization and ICU needs [as forecasted by multiple models].
PRECAUTIONARY ACTIONJUSTIFICATIONWHAT WENT WRONGStop elective procedures, delay other treatmentsFocus available resources on preparing for the COVID-19 onslaughtTreatment for major conditions like cancer were delayed, [] effective screening programs were cancelled, procedures not done on time had suboptimal outcomes
Send COVID-19 patients to nursing homesAcute hospital beds are needed for the predicted COVID-19 onslaught, models predict hospital beds will not be enoughThousands of COVID-19 infected patients were sent to nursing homes[] where large numbers of ultra-vulnerable individuals are clustered together; may have massively contributed to eventual death toll
Inform the public that we are doing our best, but it is likely that hospitals will be overwhelmed by COVID-19Honest communication with the general publicPatients with major problems like heart attacks did not come to the hospital to be treated, [] while these are diseases that are effectively treatable only in the hospital; an unknown, but probably large share of excess deaths in the COVID-19 weeks were due to these causes rather than COVID-19 itself [, ]
Re-orient all hospital operations to focus on COVID-19Be prepared for the COVID-19 wave, strengthen the response to crisisMost hospitals saw no major COVID-19 wave and also saw a massive reduction in overall operations with major financial cost, leading to furloughs and lay-off of personnel; this makes hospitals less prepared for any major crisis in the future
Seat belts cost next to nothing to produce in cars and have unquestionable benefits. Despite some risk compensation and some excess injury with improper use, seat belts eventually prevent ∼50% of serious injuries and deaths []. Measures for pandemic prevention equivalent to seat belts in terms of benefit-harm profile are simple interventions like hand washing, respiratory etiquette, and mask use in appropriate settings: large proven benefit, no/little harm/cost [, ]. Even before the COVID-19 pandemic, we had randomized trials showing 38% reduced odds of influenza infection with hand washing and [non-statistically significant, but possible] 47% reduced odds with proper mask wearing []. Despite lack of trials, it is sensible and minimally disruptive to avoid mass gatherings and decrease unnecessary travel. Prolonged draconian lockdown is not equivalent to seat belts. It resembles forbidding all commute.
Similarly, fire insurance offers a misleading analogy. Fire insurance makes sense only at reasonable price. Draconian prolonged lockdown may be equivalent to paying fire insurance at a price higher than the value of the house.
2.4. Mean, observed maximum, and more than the observed maximum
Taleb refers to the Netherlands where maximum values for flooding, not the mean, are considered []. Anti-flooding engineering has substantial cost but a favorable decision-analysis profile after considering multiple types of impact. Lockdown measures were decided based on examining only one type of impact, COVID-19. Moreover, the observed flooding maximum to-date does not preclude even higher future values. Netherlands aims to avoid devastation from floods occurring once every 10,000 years in densely populated areas [//en.wikipedia.org/wiki/Flood_control_in_the_Netherlands. [Accessed 18 June 2020]]. A more serious flooding event [e.g. one that occurs every 20,000 years] may still submerge the Netherlands next week. However, prolonged total lockdown is not equivalent to building higher sea walls. It is more like abandoning the country - asking the Dutch to immigrate, because their land is quite unsafe.
Other natural phenomena also exist where high maximum risks are difficult to pinpoint and where new maxima may be reached. For example, following Taleb’s argumentation, one should forbid living near active volcanoes. Living at the Santorini caldera is not exciting, but foolish: that dreadful island should be summarily evacuated. The same applies to California: earthquake devastation may strike any moment. Prolonged lockdown zealots might barely accept a compromise: whenever substantial seismic activity occurs, California should be temporarily evacuated until all seismic activity ceases.
Furthermore, fat-tailed uncertainty and approaches based on extreme value theory may be useful before a potentially high-risk phenomenon starts and during its early stages. However, as more data accumulate and the high-risk phenomenon can be understood more precisely with plenty of data, the laws of large numbers may apply and stochastic rather than chaotic approaches may become more relevant and useful than continuing to assume unlikely extremes. Further responses to appear in Table 5.
Table 5
Taleb’s main statements and our responses.
Forecasting single variables in fat tailed domains is in violation of both common sense and probability theory.Serious statistical modelers [whether frequentist or Bayesian] would never rely on point estimates to summarize a skewed distribution. Using data as part of a decision process is not a violation of common sense, irrespective of the distribution of the random variable. Possibly using only data and ignoring what is previously known [or expert opinion or physical models] may be unwise in small data problems. We advocate a Bayesian approach, incorporating different sources of information into a logically consistent fully probabilistic model. We agree that higher order moments [or even the first moment in the case of the Cauchy distribution] do not exist for certain distributions. This does not preclude making probabilistic statements such as P[a