by Gesine Meyer-Rath, Sheetal Silal, Juliet Pulliam and Harry Moultrie
A group of modellers recently put out a manifesto specifying “five ways to ensure that models serve society”, with specific application to the COVID-19 pandemic[i]. They summarise the uncertainty regarding almost all central aspects of how SARS CoV-2 (the virus that causes COVID-19) behaves, including its prevalence, reproduction rate, the proportion of asymptomatics, the role of seasonality, whether cross-immunity to the virus from other infections exists, and whether immunity to the virus itself persists- as well as the impact of the currently available non-pharmaceutical interventions such as social distancing and mask-wearing.
This means that making decisions regarding anything from how seriously to take the virus to what to do to contain it, is based on a high level of uncertainty; and yet, decisions have to be made by governments around the world. In this situation, the manifesto implores us that “[m]odellers must not be permitted to project more certainty than their models deserve, and politicians must not be allowed to offload accountability to models of their choosing”. Based on our collective experience of modelling COVID-19 for the South African government over the last four months, we concur. Sorting the known from the unknown, and the unknown from the unknowable, all while producing estimates that can help prevent a dangerous breaching of healthcare capacity with an ensuing dramatic increase in mortality, is what we understand to be our difficult and unenviable task.
The National COVID-19 Epi Model
Much has been written in the last weeks about the models used by government to make decisions regarding the management of the COVID-19 pandemic in South Africa. Some of this was aimed at the National COVID-19 Epi Model (NCEM), of which we are authors. The National COVID-19 Epi Model is a stochastic compartmental transmission model designed to estimate the total and diagnosed number of COVID-19 cases at provincial and district level in South Africa. The outputs of the model are used to inform resource requirements and predict where gaps could arise based on the available resources within the South African health system. The model follows a generalised Susceptible-Exposed-Infectious-Recovered (SEIR) structure accounting for disease severity (asymptomatic, mild, severe and critical cases) and the treatment pathway at outpatient level and into non-ICU and ICU care.
The model is updated regularly as new data from South Africa and evidence from countries around the world become available. We do so to conscientiously fulfil the model’s main aim- to provide relevant, timely estimates of the number of projected COVID-19 cases, including severe cases needing hospitalisation, to allow the national and provincial Departments of Health to plan for the number of people needing care. From the start, these projections have been used as inputs into a second model, the National COVID-19 Cost Model, which estimates the number of additional hospital beds, staff, and ventilators and oxygen volumes needed for inpatient care, as well as its costs- but also the likely number and costs of the PCR tests for SARS CoV-2, quarantine facilities, field hospitals, and staff at primary healthcare facilities.
All model updates are reviewed by the consortium and relevant partners at the NDOH and thereafter used in the real-time planning process. We endeavour to make the main outputs, scenarios and assumptions available to the public through the NICD website as as frequently as possible, for no other reasons than the constraints on our time. To construe this as us trying to hide something is incorrect; on the contrary, we have aimed at making all model outputs and code available for public scrutiny from the beginning.
What we got right- and what we got wrong
Another issue of public importance is the model’s degree of accuracy. There remains considerable uncertainty regarding the likely trajectory of the COVID-19 epidemic in South Africa overall, and perhaps more so due to recent flattening of the growth in cases in the Western Cape. We often hear the phrase that models are only as good as the data used to develop them. This assertion is true, though models may be inaccurate for other reasons. Models may be mis-specified if they do not account for all the variables that are key to accurately estimating the desired outcome. Additionally, the role of population behaviour in determining the trajectory and scale of the epidemic is more influential in COVID-19 due to the absence of a cure or vaccine. We do not believe that one can precisely capture human behaviour in an equation, least of all in an unprecedented situation such as the current pandemic. To account for these unknowns, we present our estimates with uncertainty bands reflective of variation in the parameters driving the model and the model process itself.
While we acknowledge that model estimates may not always be accurate, there are aspects of the epidemic that we have predicted well, such as the overall national trend in the development of cases and deaths and, most importantly, the timing of capacity breaches. Our most recent published long-term projections (6 May) predicted 822 (prediction interval: 431-1,618) deaths nationally under the optimistic scenario, and 5,486 (2,849-9,869) deaths under the pessimistic scenario by 1 July. Our short-term projections (12 June) for deaths on 1 July were 3,810 (1,880-7,270). The reported national Covid-19 deaths on the 1st of July were 2,952 which is within the range of both the short- and long-term predictions.
There are instances, however, where the development of the pandemic has taken us by surprise. Thankfully, although unexpectedly, confirmed COVID-19 deaths in the Western Cape have plateaued for the last 4 weeks, and hospital admissions for COVID-19 in the Western Cape appear to have peaked on 22 June. While the decline in admissions and plateau in mortality are welcomed, the explanation remains unclear. Did we over-estimate COVID-19 deaths in the Western Cape?
The brief answer is that we don’t know. There are at least two possible explanations for the lower than projected COVID-19 deaths in the Western Cape. Firstly, not everyone may be equally susceptible to COVID-19. While this thesis has been promoted by other groups for a few months, the evidence remains scanty. There are indications that children may be less susceptible to SARS-CoV-2 (the virus that causes COVID-19 disease) and that previous exposure to other coronaviruses could possibly provide T-cell derived immunity to SARS-CoV-2. If either or both of these mechanisms is at work, then the proportion of the population susceptible to COVID-19 will be lower than that predicted in our models, resulting in lower COVID-19 mortality. This would be good news; however, we do not yet have robust enough evidence regarding the presence and level of existing immunity to take this into account in more than a hypothetical scenario.
Secondly, public awareness of increasing deaths and the looming threat of overwhelmed healthcare facilities in the Western Cape, combined with communication campaigns, could have resulted in better adherence to NPIs (e.g. masks, hand washing and social distancing) and in those most at risk for severe COVID-19 disease taking additional precautions to isolate themselves. Since individual behaviour changes are unpredictable and difficult to model, these are not included in our models.
It is probable that the explanation is a combination of these factors. We simply don’t know yet, and it is too early to infer from the change of the Western Cape curve how the curves for the other provinces or the country overall will likely change.
The curious ‘flaw’ of regular updates
Models that incorporate and try to make sense of as much uncertainty as those estimating COVID-19 cases and associated resource use can only stay relevant and contribute to planning when they are updated as soon and as often as new evidence becomes available- be it in the shape of new local statistics or international evidence. As a result, projections differ between individual public releases and internal updates, sometimes by quite a bit. This is another “flaw” that has most recently been held against our model; we however maintain that it is our duty to update as often as we can, while taking care to communicate the reasons for each update in a transparent manner, and presenting results alongside potential factors that could change them in the future.
Among the elements that we have updated regularly is the reproductive number, both in terms of taking differences between increasingly smaller areas into account as soon as the growing case numbers allowed us to do so, and in terms of updating our assumptions about the adherence of the population to the restrictions. In past updates, we have had to decrease our assumptions regarding the reproductive number across all provinces, based on analyses taking the lower than expected number of cases into account. Possible reasons for this include, but are not limited to, better adherence to restrictions than we had initially assumed, and heterogeneity in susceptibility or mixing patterns. As a result, the projected peaks in each province have moved further apart, which allows scarce inpatient resources to go further, resulting potentially in fewer deaths from breaching the current capacity of ICU and other hospital beds and the staff that work there. This is good news with regards to the combined resources needed for the COVID-19 response as well as the required budget.
Outlook on new updates
We will continue to update our numbers, and so will the teams planning, budgeting for, and ordering resources to deal with this pandemic in each province. Amongst the next planned updates are the incorporation of mortality resulting from a potential breach of the bed capacity, the update of the inpatient care pathways in the model to incorporate non-ICU care options such as oxygen delivery via high-flow nasal cannulae, and new treatment options such as dexamethasone, remdesivir and interleukin-6 inhibitors if they are incorporated into local guidelines, as well as the additional capacity that the planned field hospitals might provide. As mentioned, we are planning to make the model code public for additional scrutiny. With these updates, we continue to strive towards making our models as useful as possible, while bearing in mind the assertion of the above mentioned manifesto that “[m]athematical models are a great way to explore questions. They are also a dangerous way to assert answers. Asking models for certainty or consensus is more a sign of the difficulties in making controversial decisions than it is a solution”.
[i] Saltelli A, Bammer G, Bruno I, et al. Five ways to ensure that models serve society: a manifesto. Nature 2020; 582(7813): 482-4.
