Imperial College London have released their Covid-19 epidemic simulation(github.com)
github.com
Imperial College London have released their Covid-19 epidemic simulation
https://github.com/mrc-ide/covid-sim
12 comments
I've seen a lot of code coming out of academia that exhibits a similar complete lack of structure and completely ignores the most basic software development best practices (such as having tests). There are exceptions, and research software engineers are a thing, but unsurprisingly most academics focus their careers on the science rather than the code.
It’s the same reason that if you go into a working Physics laboratory, most things will seem to be wrapped in kitchen aluminium foil. There’s better ways of making thermal insulation, EMF shielding, grounding, dust protection... but the foil is /right there/ and the experiment only really needs to work /once/
Hey now, how could anything be better than foil? You can reliably steal enough of it from any nearby kitchen, and it does the job well enough, what's the downside? ;)
In physics labs everything is wrapped in foil because they use heat tape to bake the shit of it.
I once had to debug a crash in a very popular program used in genomics (for those in the know, it was an early version of "samtools" IIRC) and I found it really hard to read.
(In the end the crash was due to corrupted input data, so I fixed the data and stopped debugging)
(In the end the crash was due to corrupted input data, so I fixed the data and stopped debugging)
the crash was due to corrupted input data
Indeed: in fact someone must have found the issue eventually, as newer versions exited with an error instead of core dumping.
Sadly, academic constraints (time; results required the day before) prevented me from actually filing a bug report.
Sadly, academic constraints (time; results required the day before) prevented me from actually filing a bug report.
Proper input validation is really hard, and a reliable crash on incorrect input is frankly much better than other alternatives such as soldiering on and output trash, or even worse seemingly fine results.
Of course, it sucks if it takes a few hours before you get your crash, that usually tricks you into thinking the software did something wrong and not you. Best is if it dies immediately upon reading the input, and if it does that with an error or a crash isn't crucial, though of course getting a clear error message is always better.
As a comparison, a collaboration internal tool I need to use will print a helpful usage message, and then segfault if you call it without any arguments...
Of course, it sucks if it takes a few hours before you get your crash, that usually tricks you into thinking the software did something wrong and not you. Best is if it dies immediately upon reading the input, and if it does that with an error or a crash isn't crucial, though of course getting a clear error message is always better.
As a comparison, a collaboration internal tool I need to use will print a helpful usage message, and then segfault if you call it without any arguments...
Sometimes those people found a startup company, come up with a 'prototype', and are lucky enough to be acquired.
Then, you end up having to turn this 'style' of code into a product and to maintain it...
Then, you end up having to turn this 'style' of code into a product and to maintain it...
Journals need to start rejecting submissions without code, or with code that doesn't pass peer review by people with a modicum of software engineering skill.
I was part of the GitHub team that helped get the code ready for public release. We fixed a few bugs, reduced memory consumption, made it portable across operating systems, etc., but the code you see is largely what was written by Neil Ferguson and his team.
Given your concerns, I would like to mention two things:
1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfect - the code is many years old and the module boundaries have existed for only a few weeks.
2) this was not written by professional software engineers, but epidemiologists, and they were working with limited budget, time, and programming experience.
Given your concerns, I would like to mention two things:
1) the code was originally a single source file, so I'm not surprised the module boundaries are imperfect - the code is many years old and the module boundaries have existed for only a few weeks.
2) this was not written by professional software engineers, but epidemiologists, and they were working with limited budget, time, and programming experience.
Review by someone claiming to be a ex-Googler here: https://lockdownsceptics.org/code-review-of-fergusons-model/
If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’
This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.
If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’
This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.
Epidemiologists are professional software engineers, in the sense that they are paid full time to write code and summarise the output in papers. They might not be "professional" in the sense of the quality of their work but let's not kid ourselves that these people spend half their days in the lab or taking swabs from children in Beijing. Other types of researchers do that - these guys just run sims.
Academia is geared towards paper publishing, so why would they bother with code quality? Even if in the long run it would benefit them, its at best a nice to have. Most academia doesn't share code or data anyway.
It's like saying that programmers are professional English writers, so comments and such should be written in perfect English.
IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
It's like saying that programmers are professional English writers, so comments and such should be written in perfect English.
IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
Papers should not be accepted unless they are accompanied by high quality source code.
> IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
I think the best way to tell if it is sound and implemented accurately is to first require that it is implemented as simply (to understand) as it can be. Even then, it is hard to reason about code. But without simplicity, the problem is ten times worse.
I'm tempted to go further and say that source code is formalized thought. I've written a number of simulators in the past, sometimes purely to help me understand a system more fully. There's something beautiful about describing the behaviour of a system in elegant code. When there's nothing left to remove I get much greater confidence that I've fully groked the system. I'd say this is the area where the "computers are like a bicycle for the mind" quote is the most true.
> IMHO the only valid criticism is if the the simulation modelling is sound and implemented accurately.
I think the best way to tell if it is sound and implemented accurately is to first require that it is implemented as simply (to understand) as it can be. Even then, it is hard to reason about code. But without simplicity, the problem is ten times worse.
I'm tempted to go further and say that source code is formalized thought. I've written a number of simulators in the past, sometimes purely to help me understand a system more fully. There's something beautiful about describing the behaviour of a system in elegant code. When there's nothing left to remove I get much greater confidence that I've fully groked the system. I'd say this is the area where the "computers are like a bicycle for the mind" quote is the most true.
At every place I've worked correct spelling and grammar in comments is in fact expected of developers, as is high quality documentation.
(yes I know I've been lucky)
(yes I know I've been lucky)
If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release".
The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.
The original code as used to produce simulation results that were in turn used for reports/papers/policy should be made public.
> If the point of the exercise is to allow public scrutiny of models then IMHO the code should not be changed "for public release".
Some code I've used from scientific papers in the past made a git tag at the time of publication, and said "Use version XXXX and the provided sample data to get the results in the paper", or something of the sort. This could allow review and improvements post-publication.
Some code I've used from scientific papers in the past made a git tag at the time of publication, and said "Use version XXXX and the provided sample data to get the results in the paper", or something of the sort. This could allow review and improvements post-publication.
Isn’t this a great example, why epidemiologists shouldn’t write in C++? Aren’t there some Python modules for this kind of stuff?
Edit: Oh this is very old code, that could be an excuse.
Edit: Oh this is very old code, that could be an excuse.
I appreciate you pointing out the flaws, but I fear they sarcasm is not really warranted. Shit code or not, you can call it out without denigrating the guy.
It's nice they aired their source code out for people to look at, least we can do is to critique without snark.
It's nice they aired their source code out for people to look at, least we can do is to critique without snark.
OK, fair point. Snarkiest comment removed. My comments weren't meant to be serious. The problem I pointed out isn't actually a bug, it's just a style of programming that makes the code harder to read/work on than it needs to be.
[deleted]
[deleted]
Interesting tidbit for HN, Jobn Carmack helped a little with this project apparently https://twitter.com/ID_AA_Carmack/status/1254872368763277313
Yeah and he said that code is fine so I am trusting him.
Honestly this is kind of sad. I can't understand how someone can have the gall to use any results from this spaghetti code in a scientific paper, let alone one that changes a country's entire strategy.
I'll be "that guy" and point out we know the climate simulations have similar issues.
[deleted]
[deleted]
Some people have raised issues with the model[1][2], or rather, the software implementation of it.
There's also a (flagged) submission on HN discussing this[3] referencing [1].
[1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/
[2] https://github.com/mrc-ide/covid-sim/issues/165
[3] https://news.ycombinator.com/item?id=23099212
There's also a (flagged) submission on HN discussing this[3] referencing [1].
[1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/
[2] https://github.com/mrc-ide/covid-sim/issues/165
[3] https://news.ycombinator.com/item?id=23099212
> On a personal level, I’d go further and suggest that all academic epidemiology be defunded.
Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors...
Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions.
The worst problem this all out murder attempt can muster is that the code is hard to debug which frankly is should be the default assumption for all research code, not too persuasive. Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly.
A more interesting critique would be something along the lines of this: https://www.nicholaslewis.org/imperial-college-uk-covid-19-n... (however it's not by a subject matter expert so the problems they find might well be because the misunderstood some detail).
Wow, next they'll review one doctors handwriting and conclude that hospitals should be defunded, with their job handled by horse doctors...
Non-intended randomness is of course bad, but it's bad mainly because it makes it harder to track down causes of actually important problems with the produced distributions.
The worst problem this all out murder attempt can muster is that the code is hard to debug which frankly is should be the default assumption for all research code, not too persuasive. Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly.
A more interesting critique would be something along the lines of this: https://www.nicholaslewis.org/imperial-college-uk-covid-19-n... (however it's not by a subject matter expert so the problems they find might well be because the misunderstood some detail).
If you read the lockdownskeptics cite, "hard to debug" is not the problem. Non-determinism in the output is the issue, and if this is indeed the case, why would anyone trust the results? Do a bunch of runs and average is not a good answer.
It's really not enough to say. "Do a bunch of runs and average" is exactly how quite a bit of simulation software works. In this case, a small number of random outcomes early in the "pandemic" will have a large impact on the outcome.
Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
Of course, this kind of uncertainty needs to be dealt with, and that may have been done by running the simulation code we are presented with multiple times. It may be necessary to read both the code and the associated papers to judge this correctly.
The non-determinism that page talks about is coming from bugs like memory corruptions, floating point inaccuracies, initialisation order bugs. It's not an intentional part of the model. Averaging corrupted data is meaningless, it doesn't magically fix the corruption.
I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any level of bugginess that would yield the output of these teams unacceptable, to them?
I really wonder what it would take for some people to lose faith in epidemiology. Has this field ever predicted an epidemic correctly? Is there any level of bugginess that would yield the output of these teams unacceptable, to them?
> The non-determinism that page talks about is coming from bugs like memory corruptions, ..., initialisation order bugs.
Source? I haven't seen these specifically cited anywhere.
> floating point inaccuracies,
Combined with (safe) race conditions, this will cause non-determinism that would probably be considered OK.
In general: John Carmack looked at the code and thought it was OK for what it is (decade old simulation transpiled from Fortran at some point). Some ex-Google guy thinks its horrible.
I looked at the code myself briefly. I haven't formed a strong opinion about the code myself beyond "it's ugly and I don't want to work with it, glad it's not my problem." I am however objecting to some of the comments here that make it sound like it is obviously broken for reasons that they just don't understand.
I think a comment on the GitHub is relevant: https://github.com/mrc-ide/covid-sim/issues/175#issuecomment...
> To add to this, please read report 9 properly. The 500k UK prediction was if governments did nothing whatsoever - we never believed governments would do nothing but we modelled it as a base case, because that's part of what you do when you model.
> With the full social distancing the report suggestd it might be possible to reduce deaths perhaps to 20k - a death count we have already exceeded. Nobody here is laughing about that. The report was also very frank about the uncertainty involved in trying to predict what might happen at that stage
Source? I haven't seen these specifically cited anywhere.
> floating point inaccuracies,
Combined with (safe) race conditions, this will cause non-determinism that would probably be considered OK.
In general: John Carmack looked at the code and thought it was OK for what it is (decade old simulation transpiled from Fortran at some point). Some ex-Google guy thinks its horrible.
I looked at the code myself briefly. I haven't formed a strong opinion about the code myself beyond "it's ugly and I don't want to work with it, glad it's not my problem." I am however objecting to some of the comments here that make it sound like it is obviously broken for reasons that they just don't understand.
I think a comment on the GitHub is relevant: https://github.com/mrc-ide/covid-sim/issues/175#issuecomment...
> To add to this, please read report 9 properly. The 500k UK prediction was if governments did nothing whatsoever - we never believed governments would do nothing but we modelled it as a base case, because that's part of what you do when you model.
> With the full social distancing the report suggestd it might be possible to reduce deaths perhaps to 20k - a death count we have already exceeded. Nobody here is laughing about that. The report was also very frank about the uncertainty involved in trying to predict what might happen at that stage
Exactly, the individual runs aren't actually the "deliverable" of the code; rather it is the average of many runs that represent the real result of the code.
Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear.
To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.
Nothing presented clearly compromised the (supposed) reliability of the distributions produced, so the impact of these bugs beyond the inconvenience they add is unclear.
To be clear, it is certainly not true that removing these bugs will somehow prove that the model and its inputs themselves are correct.
> Models are after all just tools: what is critical is that you've made reliable predictions, not that the tools themselves are easy to use correctly.
Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow.
That said, that "personal level" statement is absolutely out of the line.
Reliable and reproducible. I think this discussion, even if the code ends up being correct and the model OK, is worth having. It might not change anything today, but could set new (hopefully better!) standards tomorrow.
That said, that "personal level" statement is absolutely out of the line.
Sure, but for an inherently stochastic model the main results are the average behaviour, and the reproducibility of these are the main concern. Reproducibility of individual runs is more a convenience.
That issue is unhelpful. Yes, no good tests, but how about suggesting ways it can actually be improved? Why are devs such bores when it comes to things like this? They have released the model, review it and suggest improvements! Don't grandstand "We, the undersigned etc etc" as if that's going to help improve the codebase in the slightest.
Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack.
EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.
Carmack is OK with it, he's put his name to reviewing it. That's not to excuse the bad testing, but he hasn't thrown his hands up and run away. He worked with them constructively to make it possible for us to even see it! Be more like Carmack.
EDIT: Also be like this person: https://github.com/mrc-ide/covid-sim/issues/161 Helpful, constructive and the developers engaged with them.
Carmack has different motivations. He has nothing to prove and simplistically looks to make things better. You can tell in his appearance on Joe Rogan.
Most HN users are just publicly preening. It's like a Mechanical Turk GPT2. I actually doubt they can write code.
Most HN users are just publicly preening. It's like a Mechanical Turk GPT2. I actually doubt they can write code.
I fear that this area is too divisive currently to allow for normal discourse. The downvoting in these kinds of topics have been atrocious on hacker News the last few weeks.
HN is one of the increasing fewer places online where covid19 deniers can congregate.
I personally don't like much the term "deniers", because there are a lot of unknowns on this virus and what it does, and there is no agreement on many fronts.
Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism.
Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match).
I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.
Also, doing these generalizations groups together people with very questionable theories ("It's the 5G") with others that have more nuanced criticism.
Personally (and yes, I am a scientist) try to look up whatever is said in the media, either by journalists or experts, no matter if the results end up matching 100% what it is said (often it is less, and on some cases there is no match).
I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies.
I started using the term deniers because all of them are studiously ignoring something that doesn't pencil out. It doesn't matter if they are loony deniers, sciency deniers, I have a big brain tech bro deniers. The result is all the same. Garbage.
Dangerous garbage.
Dangerous garbage.
It's too late. The early buggy code was already used to drive decisions, and they refused to release it for a month and a half whilst they tried to fix the most embarrassing errors.
Go click around GitHub. They screwed up a shuffle, there are uninitialised reads, RNG bugs, the works.
Go click around GitHub. They screwed up a shuffle, there are uninitialised reads, RNG bugs, the works.
If this has been flagged off the front page I despair for HN. This should be an apex story for us: open source release of a computer model that has incredible relevance to loads of people globally. Quite sad.
I'm pretty sure it was. It was at number four and rising then suddenly dropped to the bottom of the front page before vanishing.
I'm no expert in C++ so not sure if it's meant to look like that, but that code is making my brain hurt.
Just argument parsing makes my eyes bleed...
https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi...
That stuff is priceless:
else if (argv[i][1] == 'C' && argv[i][2] == 'L' && argv[i][3] == 'P' && argv[i][4] == '1' && argv[i][5] == ':')
I guess string comparisons are complicated. I also fail to see why they used ':' as the separator and why they didn't use a proper library to parse argv...
https://github.com/mrc-ide/covid-sim/blob/master/src/CovidSi...
That stuff is priceless:
else if (argv[i][1] == 'C' && argv[i][2] == 'L' && argv[i][3] == 'P' && argv[i][4] == '1' && argv[i][5] == ':')
I guess string comparisons are complicated. I also fail to see why they used ':' as the separator and why they didn't use a proper library to parse argv...
Because when you have only a very limited number of arguments it’s easier to do it that way than selecting a lib, adding it to the project, reading the doc, trying it and integrating it for real. I add the issue in C#, and after trying 2 or 3 libs (including one by Microsoft) I just gave up.
I hear you but in their case they have more than 130 LOC to parse ARGV. Doing all that manually hurts both code readability and maintainability.
Having seen a discussion about this on a different forum, someone who used to work in academia mentioned that repeatability and consistency are key to academic code, which means often readability and convention may get sacrificed - it works for the author and if someone wants to repeat the experiment it should get the same results without wangling around with many external packages or modules that may change or become obsolete.
Still hurts to read though :P
Still hurts to read though :P
Yep. Also, speaking from academic point of view, code won’t make you advance on your career by itself (only papers based on it will), so the minimum is done until the thing is usable for its goal. So forget about CI/CD, proper test suits, documentation (I hate this point personally and document my projects a bit more than the average researcher) and engineering best practices.
Then public policy should heavily discount results based on academic code, if it shown to be poorly engineered. I'd go further and say you can't trust the papers based on the results of badly engineered simulations. As a poster said earlier "I think there should be fairly high standards of scientific rigor even in published code, especially if this might impact public policy actions, like we should expect high rigor in biological and epidemiological studies."
...what do you think those biological and epidemiological studies hinge on? ;)
it is not meant to look like that. The spacing/tabbing is all over the place.
'Academic code' is infamous for having that effect.
Alumnus here, it’s Imperial College, London. It is a university (in its own right), but always goes by that name.
I knew that; I went to Kings! Apparently I haven't woken up properly this morning. Title changed.
Alum of both, it's comma-less icl on paperwork, website, rankings, and research, see ie https://www.imperial.ac.uk/mrc-global-infectious-disease-ana... bottom their address.
Ah but I somehow managed to write "Imperial University". I have now had a coffee, and changed the title.
Only since 2007. Uni of London before that for 100 years. Used to play them at sports as part of the ULU league.
No, Imperial's royal charter (what's needed to be 'a university' and award degrees in your own right) dates to 1907. I believe that's true of most of UoL's constituents, just not the smaller ones perhaps like SOAS or other field-focused colleges.
Yes it was part of the University of London, but always “Imperial College”. As far as I’m aware never “Imperial University” as the title initially had it here.
Individual colleges were effectively independent universities for many years before. The University of London umbrella was a shell.
I guess confidence really is key when making code public...
[deleted]
For a long time academics would only publish formulas and results as part of their papers - omitting source code and arguing that ‘the formulas are the only thing that matter’.
Perhaps the real reason is the code typically smells of Swiss cheese.
From now on I vow to trust NO academic results of computer models unless source code is published along with instructions on how to reproduce outputs (or at least similar output!)
Perhaps the real reason is the code typically smells of Swiss cheese.
From now on I vow to trust NO academic results of computer models unless source code is published along with instructions on how to reproduce outputs (or at least similar output!)
Is this thing actually being used for policy decisions?
Given the complexity, poor language/framework choice (should have used either Rust or Tensorflow), bad code design, and unclear determination of the parameters (especially the fact they don't seem to be estimated from real data with Bayesian inference, or if they are they didn't release that), as well as the ludicrous CPU and RAM usage making it impossible for most people to run it and thus check it, it doesn't seem like it has any chance of actually being a good model.
If this is the state of the art for the most important statistical modelling project in the world, we are in the dark ages.
Given the complexity, poor language/framework choice (should have used either Rust or Tensorflow), bad code design, and unclear determination of the parameters (especially the fact they don't seem to be estimated from real data with Bayesian inference, or if they are they didn't release that), as well as the ludicrous CPU and RAM usage making it impossible for most people to run it and thus check it, it doesn't seem like it has any chance of actually being a good model.
If this is the state of the art for the most important statistical modelling project in the world, we are in the dark ages.
It's 13+ years old. There was no Rust or Tensorflow when it was originally produced.
13+ years ago the same academic modeler predicted 200 million deaths from a bird flu outbreak that killed 282 people in total.[1]. That's a pretty enormous error-- alarmist publicity-seeking behavior then and now, but the damage was felt more acutely now. If this modeler really wanted a competent simulation it would have been better constructed. That it was not well-constructed is another datapoint indicating how egregious the error was on the part of the goverment in failing to properly vet it.
[1].https://www.express.co.uk/comment/columnists/frederick-forsy...
[1].https://www.express.co.uk/comment/columnists/frederick-forsy...
That's an amazingly disingenuous comment.
https://www.theguardian.com/world/2005/sep/30/birdflu.jamess...
He said that if it was the same as the 1918 flu, you could probably scale it up to 200M people. He didn't predict 200M, he speculated that it could be that bad.
And he wasn't alone - "A global influenza pandemic is imminent and will kill up to 150 million people" said "David Nabarro, one of the most senior public health experts at the World Health Organisation"
"A Department of Health contingency plan states anywhere that there could be between 21,500 and 709,000 deaths in Britain."
An unnamed WHO spokeswoman said "best case scenario" would be 7.4 million deaths globally.
https://www.newscientist.com/article/dn7787-flu-pandemic-let...
"And yet, the models show, if targeted action is taken within a critical three-week window, an outbreak could be limited to fewer than 100 individuals within two months."
Oh, he also showed how to react and keep the number of casualties to a minimum.
Don't believe what you read in dirt rags.
https://www.theguardian.com/world/2005/sep/30/birdflu.jamess...
He said that if it was the same as the 1918 flu, you could probably scale it up to 200M people. He didn't predict 200M, he speculated that it could be that bad.
And he wasn't alone - "A global influenza pandemic is imminent and will kill up to 150 million people" said "David Nabarro, one of the most senior public health experts at the World Health Organisation"
"A Department of Health contingency plan states anywhere that there could be between 21,500 and 709,000 deaths in Britain."
An unnamed WHO spokeswoman said "best case scenario" would be 7.4 million deaths globally.
https://www.newscientist.com/article/dn7787-flu-pandemic-let...
"And yet, the models show, if targeted action is taken within a critical three-week window, an outbreak could be limited to fewer than 100 individuals within two months."
Oh, he also showed how to react and keep the number of casualties to a minimum.
Don't believe what you read in dirt rags.
Metawards is also being used.
https://github.com/ldanon/MetaWards
https://github.com/ldanon/MetaWards
There is also a more user friendly version of the MetaWards at https://metawards.org/ ( GH repo: https://github.com/chryswoods/MetaWards ) implemented in Python which identically matches the results of the C implementation, verified by tests.
I like the way InitModel() crashes (I think) if a global pointer called bmh (short for bitmap header) isn't first initialized by calling InitBMHead() from Bitmap.cpp. I guess it's obvious to academics with giant brains that InitModel() depends on a bitmap existing.
But it gets worse - the pointer is declared in CovidSim.cpp but Bitmap.cpp reaches across the module boundary to initialize it. Sweet.
Edit: I feel bad now. It's great that this code was made public. I should be encouraging that. I think academics writing code for things like this should probably take an undergrad course on software engineering. I think the one that taught me about Abstract Data Types would be most beneficial the author(s) of this code.