COVID-19 modelling is wrong
Compartmental models used by some governments to model COVID-19 critically underestimate our ability to control the epidemic
Dr James Jansson PhD
CrabMe Pty Ltd Modelling Group
James Jansson has PhD in Mathematical Modelling of HIV obtained from the University of NSW, Sydney Australia.
Disclosure: James Jansson is the Deputy Leader of the Science Party (Australia).
Abstract
Governments are currently using SEIR compartmental models that are inadequate at accurately reflecting the interactions between people, the disease progression in those who are infected and the possible strategies towards elimination. While it is likely that these models are sufficient to model the epidemic in the early stages, they struggle as the network of connections between people become saturated with recovered people, and incorrectly model the recovery process which is critical in demonstrating if quarantining measures work.
Compartmental models give an overly pessimistic view of the likelihood of success of short, sharp intervention strategies that could see total clearance and society return to normal sooner.
This paper demonstrates the difference between compartmental models and agent-based models, the limitations of compartmental models in general and demonstrates the limitations the CovidSim that New Zealand (and potentially other countries) are using as the basis of their modelling.
Introduction
With the release of COVID-19 modelling by the New Zealand government on 31st of March 2020 and the release of the Australian model on the 7th April 2020, the public (and academics with training in the field), finally can critically analyse the assumptions and methods used to model the epidemic and its projections.
The model that many governments have used, a compartmental SEIR model (Susceptible, Exposed, Infected, Recovered model), is a thoroughly flawed type of model when it comes to modelling of epidemics, particularly when trying to apply behavioural interventions. The New Zealand government used the covidsim.eu compartmental SEIR model, the Doherty Institute created a compartmental SEIR model for the Australian government and Prof Neil Ferguson at the Imperial College initially produced a compartmental SEIR model for informing the UK government.
The inability to faithfully model inter-person dynamics and the recovery of infected individuals make these compartmental SEIR models inappropriate when making decisions involving questions about how to arrest the spread of COVID-19.
The difference between Compartmental and Agent-based models
While compartmental models were historically useful when computational power was limited (or where mathematicians solved these equations by algebraic means), even relatively modest modern computers can run more complicated agent-based models that give more accurate results. To simplify the discussion, I will be comparing a simpler compartmental SIR model (Susceptible, Infected, Recovered model), to a simple agent-based SIR model. I have produced a simple piece of software that demonstrates, in principle, how these models work and how it affects outcomes in the model. This software can be found at http://covidagentmodel.com/.
Important note: These results should NOT be taken as indicative of what is likely to happen in this COVID-19 outbreak. The models are too simple and do not consider enough complexity. If you are a government or organisation that wants to use agent-based models, please contact me at contact@covidagentmodel.com and a model that suits your particular requirements can be developed.
Compartmental models
Compartmental models work by having only a few variables representing the state of the system. This simple model is initiated with three numbers: susceptible, infected and recovered. To begin with, the majority of individuals are in the susceptible compartment, a small number of individuals are in the infected compartment, and there are no individuals in the recovered compartment. At each step some of the susceptible compartment are moved into the infected compartment, and some of the infected compartment are moved into the recovered compartment.
Movement from susceptible to infected is proportional to both the total number of people currently infected multiplied by the fraction of people who are currently susceptible. To put this plainly, if you have twice as many people infected currently, they can infect twice as many people, however as more and more people get infected, the number of people an infected person can infect goes down.
Movement from infected to recovered is proportional the number currently infected. In plain English, at each step some of the people who are currently infected move into the recovered compartment; the more people in the infected compartment, the more people move into the recovered compartment each step.
Agent-based models
Agent-based models are much more complicated and hence can be crafted to better match the reality of the epidemic being simulated than compartmental models. Agent-based models keep track of much more information about the system that is important to infection spread and recovery. Each individual is stored separately, and the state of each individual is updated independently. That means that infection dynamics like the infection burning out in a local a local area is possible.
Compartment models are typically (but not always) deterministic. That is, the previous step perfectly informs what happens in the next step. Agent-based models tend to be stochastic, where infections between individuals occur on a probability basis, much more like what happens in real life.
As the state of every individual is stored separately, it also means that the start of the infection and the time until recovery can be stored separately. This gives a much more accurate view of the path to elimination, as described in the next section.
Simulating recovery of infected individuals
Recovery in the compartmental model
In this compartmental SIR model, we need to model the time until infection. Let’s assume that the median time until recovery is 14 days. In the above system, we can determine b by starting the system with only people in the infected compartment, then adjusting the constant b until 50% of the people in the infected bucket have moved into the recovered bucket. In this case, setting b to 0.048 results in a median time until recovery of 14 days.
Recovery in the agent model
In our agent model, we used a very simple method to simulate infection recovery: when a person is infected, we assign a time until recovery of 14 days plus a random value between -7 and 7, which gives an equal distribution around the 14 day mark.
Comparing the results
The difference between the two models are striking. These differences have important impacts on the outcome of an epidemic. In the compartment model, at each day 4.8% of people are moved from infected to recovered. Unfortunately, this means it has this big, long tail, because the proportional reduction in infections per step is an exponential decay. This tail never goes to zero. This long tail is not representative of reality. After 50 days there should be essentially no-one who is remaining as infected.
Comparing the compartmental to the agent-based model, we can see that there is a limit to how long a person is expected to be infected in the agent-based model. This is an important part of epidemiology and fundamental to interventions such as quarantine: we expect all people will either recover from their infection or die and then they can be released from quarantine.
It should be noted that this distribution of recovery can be any shape you desire it to be in an agent-based model. While this distribution looks artificial, this is simply to demonstrate that we can have more conditions on the time until recovery, such as a minimum and maximum time until recovery.
This shows the true advantage of agent models over compartmental models. When a susceptible person is moved into the infected compartment, we do not keep accurate information about when they moved into that infected compartment and hence when they should move into the recovered compartment. Agent-based models do keep this information.
Simulating an uncontrolled epidemic in these basic models
So how do the models perform when you simulate what may happen if you let the epidemic rip through the community? Both the compartmental and agent models were started with 100,000 people in the model, and 100 initial infections. The parameters were adjusted the such that the epidemic doubles every three days in the early stages, and the intervention slider was set to zero.
Both epidemics had similar outcomes: most people were infected, the peak infected was at the 37 day mark, and most people recovered by 100 days. However, there was a very important difference: by day 100 in the compartmental model, there were still many people who were infected in the model, whereas in the agent model there were no infected remaining, and some people never got infected.
These differences are due to the excessively long and fat tail of the recovery profile of the compartmental model, and the fact that in the agent-based model, the infection can become locally burnt out providing herd immunity to some people as their peers are no longer susceptible.
The difference between these models become even more stark when modelling an intervention.
Simulating a 90-day, 90% reduction in contacts intervention
Using the settings from the controlled epidemic simulation previously, the settings were updated to start an intervention at the 20 day mark, reducing connections to just 10% of its previous levels for 90 days.
In both models, the number of infections decrease, however the compartmental model never reaches zero infected individuals, meaning that once the intervention is lifted, the epidemic bounces right back again. The agent-based model, however, successfully reduces the number of infected to zero which means that lifting the intervention does not result in a new outbreak.
The interesting thing about the compartmental model is that no matter how long you make this type of intervention, you always have at least a small number of people in the infected compartment, meaning that when you lift the intervention, you will always get a rebound in the number of infected.
Demonstrating the inadequacy of New Zealand’s model choice in modelling behaviour interventions
The model used by the New Zealand government is the http://covidsim.eu/ model which is publicly available. The model is clearly a sophisticated SEIR model, however it still struggles to adequately model intervention strategies and the possibility of complete elimination.
To demonstrate these severe limitations, I used extreme behavioural intervention settings in the model to show that SEIR models, including sophisticated models, are incapable of accurately modelling the epidemic and hence strategies that may lead us to eradication. Starting with a population of 10 million people and 100 people initially infected, I allowed the model run with no intervention for 60 days. At the 60-day mark, with 635,131 people infected, I turned on the intervention: for 90 days there will be a 99% reduction in contact between people.
It would be rational to predict that if all people were completely isolated from all other people, that the virus would be completely eliminated after all infected individuals recovered. Unfortunately, this is not the case.
Inspecting the graph closely reveals something very peculiar: the graph appears to report the epidemic going to zero. New infections should be coming from the infected compartment. It turns out the infected compartment does NOT contain zero people. It actually contains some fraction of a person, say 0.1 people which, when you lift the restrictions, allow the population to become reinfected. As such, the model never achieves elimination of the virus from the community.
Why does this model make such a poor prediction? It is because compartmental models suffer from fatal flaws due to oversimplification, in particular:
1) Disease progression: Compartmental models do not take into account the progression of the disease in an individual over time, as it treats the infected pool as one large group, thereby losing important temporal information about new infections as they are added to the ‘infected’ subtotal.
2) Connections between people: Compartmental models do not take into account the connectivity of the people which limits understanding about how behavioural interventions can work. They do not consider the disease spread is a probabilistic process, meaning that elimination is never possible
Implications
The implications for policy makers are enormous. When given a dire prediction, like “no matter how much we self-isolate, this virus will always come back” politicians will consider locking people down for years to be a reasonable option. However, the dire predictions made by compartmental models hide morally acceptable, short-term interventions: short, sharp interventions to eliminate infection from the population.
It is my opinion that the current lock-down, especially in isolated countries like Australia and New Zealand, can potentially completely eliminate the infection and maintain that elimination as long as all international visitors are sufficiently isolated. We can potentially speed up this elimination by making further adjustments such as requiring face masks in public to ensure mini breakouts don’t happen among critical healthcare and service workers.
Impact on economic modelling
If the epidemic modelling delivers poor results, then it will provide an incorrect basis for business decisions and the necessary stimulus and welfare packages. While I would caution against relying too heavily on modelling, agent-based modelling provides a hope that we previously did not have; we can eliminate COVID-19 from our community, and the pain would be measured in months, rather than years. This can give both business and citizens the strength and determination to stick to the lockdown and be ready to exit the other side with all pistons firing.
Model selection emphasises transparency during disasters
I do not fault the governments of the world acting decisively on the basis of the modelling provided so far or the actions taken. Fast action is an absolute necessity and interventions such as isolation can buy society time until we determine the best path forward.
The main issue that I see is that this modelling was kept secret for far too long. Once it was evident that serious action was necessary by all people in society, the models used to inform those choices should have been released. More academics, myself included, could have contributed to the formation of better policies in the face of this common threat.