The New York Times Releases Its Dataset of U.S. Confirmed Coronavirus Cases(nytco.com)
nytco.com
The New York Times Releases Its Dataset of U.S. Confirmed Coronavirus Cases
https://www.nytco.com/press/the-new-york-times-releases-its-dataset-of-u-s-confirmed-coronavirus-cases/
162 コメント
Noob Question: Anyone know an easy way to convert FIPS data (that this nytimes dataset uses) to Postal code?
It's a good question, and unfortunately, the answer is that zipcode does not map well to other kinds of geographical regions, or to day-to-day-reality for that matter.
To directly answer your question, it is not uncommon for zipcodes to span county boundaries. And there are even a handful that span more than one state: https://gis.stackexchange.com/questions/53918/determining-wh...
To the broader question of why zip codes can be lacking for geospatial analysis, here's a real-world situation:
https://theconversation.com/how-zip-codes-nearly-masked-the-...
Here's a more broader essay from CartoDB: https://carto.com/blog/zip-codes-spatial-analysis/
To directly answer your question, it is not uncommon for zipcodes to span county boundaries. And there are even a handful that span more than one state: https://gis.stackexchange.com/questions/53918/determining-wh...
To the broader question of why zip codes can be lacking for geospatial analysis, here's a real-world situation:
https://theconversation.com/how-zip-codes-nearly-masked-the-...
Here's a more broader essay from CartoDB: https://carto.com/blog/zip-codes-spatial-analysis/
Interestinggg. So, as mentioned by another user, I'm better off mapping Lat/Lon to FIPS. Found a census endpoint that does just that[1]!
Now for an address, I can geocode it up.
https://geo.fcc.gov/api/census/
Now for an address, I can geocode it up.
https://geo.fcc.gov/api/census/
You can do better than geocoding: use one of the county-level shapefiles from the TopoJSON project: https://github.com/topojson/us-atlas
At the very least, if you do just need point-level info, you can get the centroids of the county shapes.
At the very least, if you do just need point-level info, you can get the centroids of the county shapes.
I don't think it's generally possible. The FIPS code just identifies the county (the first two digits are the state, e.g. 06 for California) and the final three identify a county in that state. So technically that column is redundant with the "county" and "state" columns; I suspect they've included it to make joining this dataset to other data that might use different name-formatting of counties/states easier.
ZIP/postal codes are generally smaller (in the county I live in there are almost a hundred ZIP codes). I'm not sure they're even guaranteed to be entirely within one county either. We tend to think of ZIP codes as boundaries but they're actually delivery routes (which if you squint can be converted into boundaries by joining the properties that those routes serve together).
You might be interested in this article: https://carto.com/blog/zip-codes-spatial-analysis/
ZIP/postal codes are generally smaller (in the county I live in there are almost a hundred ZIP codes). I'm not sure they're even guaranteed to be entirely within one county either. We tend to think of ZIP codes as boundaries but they're actually delivery routes (which if you squint can be converted into boundaries by joining the properties that those routes serve together).
You might be interested in this article: https://carto.com/blog/zip-codes-spatial-analysis/
zip codes span counties and also states. A few are actually in three different states.
(I have suffered at the hands of zip-codes-as-geographic-entities)
(I have suffered at the hands of zip-codes-as-geographic-entities)
Right, so a simple mapping I'd want is given a Zip Code, what is the corresponding FIPS Code.
Pick a lat/long, and you can map between. Or find the centroid of the zip code and map accordingly. Or you might consider unions between the geometry objects. Qgis3 has a lot of great packages for this.
problem is somewhat ill-posed, but see
https://raw.githubusercontent.com/bgruber/zip2fips/master/zi...
https://raw.githubusercontent.com/bgruber/zip2fips/master/zi...
It's nontrivial.
Check out ZCTA crosswalk in US census.
Check out ZCTA crosswalk in US census.
Oh boy, I can hear the the bootcamp data scientists firing up their medium blogs from here.
Let's try to not put people down for self learning. Knowledge can be found anywhere. Those that mock people like this will soon find themselves left behind with ancient university based education. Unless you plan to re-enroll every 10 years the world will pass you by if you are not self learning.
As someone who learnt data science almost entirely by self-learning (and getting eviscerated by HN whenever my work hit the front page), public datasets were an immensely important tool for prototyping and refining my data analysis skills.
However...
The coronavirus is a case where it's very easy to unintentionally misinterpret the data, and casual analysis will likely add more noise/false information which is not what is needed now. (and any analysis will become out-of-date very quickly).
However...
The coronavirus is a case where it's very easy to unintentionally misinterpret the data, and casual analysis will likely add more noise/false information which is not what is needed now. (and any analysis will become out-of-date very quickly).
I agree. If there's been a recurring theme in every single data-touching class in my DS major, it's been ethics.
Now, if we could only get SE and DS professionals to stop working on mass surveillance and population manipulation systems.
I'm happy to hear that. Ethics is oft-neglected, but I believe it's imperative that we guide our efforts using it.
[deleted]
Agreed, it's an arrogant and condescending comment that does not belong on HN let alone at the top. It's sad.
> arrogant and condescending
Arrogant and condescending don’t mean “wrong”. Djikstra (the king of accusations of arrogance), once pointed out that his native language has no word for “egghead”: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD12xx/E..., principally because his native culture has no history of dismissing the educated. OP is using (I interpret) “bootcamp data scientists” as a shorthand for people who know far less than they think they do. That seems like a fair characterization, honestly: that’s entirely the philosophy behind bootcamps in the first place.
The idea behind them is that a four-year education in computer science is unnecessary for routine software development jobs, and one can be trained up to useful programming ability in a much shorter time (and less cost) than a full-blown university education takes. While it’s difficult to say how true this is, OP (I interpret) and I have both observed that bootcamp types seem to presume that a few-month intensive bootcamp experience actually does cover everything that a four-year degree does, and are often surprised (as well as defensive) when they come across something that their bootcamp didn’t prepare them for.
A close analogue would be a paralegal thinking they know the law better than lawyers or nurses thinking they know medicine better than doctors - not that they don’t know a lot, but there are a few things that, while not as commonly useful, do come up in actual practice that are important when they do.
Arrogant and condescending don’t mean “wrong”. Djikstra (the king of accusations of arrogance), once pointed out that his native language has no word for “egghead”: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD12xx/E..., principally because his native culture has no history of dismissing the educated. OP is using (I interpret) “bootcamp data scientists” as a shorthand for people who know far less than they think they do. That seems like a fair characterization, honestly: that’s entirely the philosophy behind bootcamps in the first place.
The idea behind them is that a four-year education in computer science is unnecessary for routine software development jobs, and one can be trained up to useful programming ability in a much shorter time (and less cost) than a full-blown university education takes. While it’s difficult to say how true this is, OP (I interpret) and I have both observed that bootcamp types seem to presume that a few-month intensive bootcamp experience actually does cover everything that a four-year degree does, and are often surprised (as well as defensive) when they come across something that their bootcamp didn’t prepare them for.
A close analogue would be a paralegal thinking they know the law better than lawyers or nurses thinking they know medicine better than doctors - not that they don’t know a lot, but there are a few things that, while not as commonly useful, do come up in actual practice that are important when they do.
I've worked as a programmer for 15 years. I did a Philosophy degree.
I can run circles around some programmers with a CS degree.
CS degrees have always been an odd qualification, mainly because so much of it is completely unrelated to professional programming. You can get a high mark and still be unable to write a non-trivial program on your own.
When I originally started doing my Chemistry degree, we did huge amounts of lab work compared to the odd class or two CS students did.
You don't come out of uni a good progammer if you study CS, while you will come out of uni a good chemist, biologist or engineer.
And that really sums up the problem with this line of argument.
I can run circles around some programmers with a CS degree.
CS degrees have always been an odd qualification, mainly because so much of it is completely unrelated to professional programming. You can get a high mark and still be unable to write a non-trivial program on your own.
When I originally started doing my Chemistry degree, we did huge amounts of lab work compared to the odd class or two CS students did.
You don't come out of uni a good progammer if you study CS, while you will come out of uni a good chemist, biologist or engineer.
And that really sums up the problem with this line of argument.
There are individuals who on their own can get all the education they need from free public resources. These are the people who can get a college degree's worth of education out of a public library on their own.
They represent an infinitesimal minority of the population. Most people, even people with CS degrees lack that discipline. At least the CS degree forces them to learn some fundamentals so that they can potentially understand what they're looking at and learn on the job, and demonstrates at least a moderate interest in the subject as a vocation.
In my experience most people who graduate from bootcamps (in general, some are better than others) are all about getting a job as soon as possible. They're expecting a 6 figure salary within a year, and are willing/able to put in 70+ hour weeks for a few months to get there. Not knocking that perspective, but they'll lack a lot of background outside of their very specific niche and will likely be less adaptable, particularly on any highly technical subjects where some level of theory is required.
IMO the wave of "Software Engineering" degrees that are trickling out into the universities are probably the ideal. Full 4 year accredited degree that focuses on practical software development and less on the theory and mathematics. Let the CS-degrees focus on research/academics in the fashion of schools of Arts and Science and leave the Engineering to the Engineering schools.
They represent an infinitesimal minority of the population. Most people, even people with CS degrees lack that discipline. At least the CS degree forces them to learn some fundamentals so that they can potentially understand what they're looking at and learn on the job, and demonstrates at least a moderate interest in the subject as a vocation.
In my experience most people who graduate from bootcamps (in general, some are better than others) are all about getting a job as soon as possible. They're expecting a 6 figure salary within a year, and are willing/able to put in 70+ hour weeks for a few months to get there. Not knocking that perspective, but they'll lack a lot of background outside of their very specific niche and will likely be less adaptable, particularly on any highly technical subjects where some level of theory is required.
IMO the wave of "Software Engineering" degrees that are trickling out into the universities are probably the ideal. Full 4 year accredited degree that focuses on practical software development and less on the theory and mathematics. Let the CS-degrees focus on research/academics in the fashion of schools of Arts and Science and leave the Engineering to the Engineering schools.
A famous Dutchman once opined that a CS degree has as much to do with teaching programming as an astronomy degree has to do with teaching the construction of telescopes.
Not sure what CS degrees you're talking about, but having done both a Biochemistry and a CS degree as an undergraduate, your description doesn't comport with what I've seen.
I had on average about one lab class a semester as a Biochemist, and on average one programming class a semester as a CS. You couldn't pass the CS class without being able to write code that worked and got more complicated by the end of the semester. By no means did this mean I (or anyone else) was an awesome software engineer by virtue of doing the degree, but I'd put almost anyone with a decent grade from my undergrad program into an entry level position. Same deal with Chemists or Biochemists. I've done a bunch of hiring and can say often bootcamp grads are less able to come up with a sophisticated answer to a problem they haven't studied before.
I say this also as a grad of Insight, a DS bootcamp of sorts.
I had on average about one lab class a semester as a Biochemist, and on average one programming class a semester as a CS. You couldn't pass the CS class without being able to write code that worked and got more complicated by the end of the semester. By no means did this mean I (or anyone else) was an awesome software engineer by virtue of doing the degree, but I'd put almost anyone with a decent grade from my undergrad program into an entry level position. Same deal with Chemists or Biochemists. I've done a bunch of hiring and can say often bootcamp grads are less able to come up with a sophisticated answer to a problem they haven't studied before.
I say this also as a grad of Insight, a DS bootcamp of sorts.
mattmanser(2)
Forget about CS degree, I once worked with a CS postdoc, now a professor at a top US university, who prided himself of never having written a program.
> You don't come out of uni a good progammer if you study CS
Do you think that this is untrue of boot camps?
Do you think that this is untrue of boot camps?
Well, no, but it's roughly the same amount of time you'll spend on learning programming in a CS degree.
And you're not going to be able to pass it because you can pontificate at length about compiler design or explain what a linked list is, but flunked the practicals because none of your code even compiled.
One guy I worked with a long time ago that we hired would give us all these huge lectures about the right DB design and when to use structs or classes. Had an opinion on the right way to do everything. But he didn't seem to be clearing many bugs, and then our company fired him after 3 months when they realized in his main project to get our dropbox-esque system talking with Word he'd written a whole 10 lines of code.
Could talk the talk, but couldn't walk the walk.
And you're not going to be able to pass it because you can pontificate at length about compiler design or explain what a linked list is, but flunked the practicals because none of your code even compiled.
One guy I worked with a long time ago that we hired would give us all these huge lectures about the right DB design and when to use structs or classes. Had an opinion on the right way to do everything. But he didn't seem to be clearing many bugs, and then our company fired him after 3 months when they realized in his main project to get our dropbox-esque system talking with Word he'd written a whole 10 lines of code.
Could talk the talk, but couldn't walk the walk.
Well, the problem isn't that the bootcamp people lack a computer science education, it's that they lack any formal education. I've hired epistemologists and physicians to program computers and it turned out fine. The bootcamp process might in fact work well as training for otherwise educated people.
It is a joke and social critique and if someone takes it seriously then that's their problem.
He's not putting down people for self-learning, he's putting down people who think that they're an authority on a subject because they put in a few 70 hour weeks in a classroom.
I understand that sentiment. But at the same time, if it ain't misinformation, I'm not sure what's wrong here. Sometimes the way to get started is by writing stuff while you are still too arrogant to not doubt (said with much affection). From code to essays, people's pasts are littered with stuff they cringe at years later. But it is what gets them started. So yea. Maybe we don't put anyone on a pedestal for shiny charts. But maybe we don't tear them down either :)
(also I say this fully appreciating that you were explaining the parent poster's view and you may not necessarily support it)
(also I say this fully appreciating that you were explaining the parent poster's view and you may not necessarily support it)
The issue is that making a mistake here isn't like making a mistake about, say, movie review sentiment analysis. Already there have been a few "I know stats!" posts that didn't do the right controls but still got sent around the internet as proof of someone's argument (see the "Viral advertising exec now knows epidemiology" post that medium took down). These data are confusing and difficult. It turns out there's a bunch of people who have studied this field for years and can help us avoid those issues (epidemiologists). We should listen to them. That's not to say we shouldn't open the data, but handle with extreme care, which isn't the modality of most early data scientists. A 4 week bootcamp doesn't lead to the most rigorous science, usually that's not a big issue (Pareto principle and all that), but when it's lives and active problems, it can become a big issue.
I say this as a graduate of Insight which is a kind of DS bootcamp.
I say this as a graduate of Insight which is a kind of DS bootcamp.
No! We should rigorously scrutinise epidemiologists and do lots and lots of competing analysis.
What makes you think epidemiologists are so great? The one who has guided the UK's response has been severely criticised by fellow academics in the past for bogus low quality modelling. The Imperial College paper has basic errors and flawed assumptions that are obvious even to untrained laymen.
I think you're wildly over-estimating how much statistical and logical training most academics get. This is one of the basic underlying problems that the replication crisis has exposed: an unending flood of academic papers, especially those doing modelling, that fall apart when examined by people with statistical and mathematical training.
You're right that a 4 week bootcamp won't make someone a flawless handling of data. But it might still be a more rigorous form of training than epidemiologists get.
What makes you think epidemiologists are so great? The one who has guided the UK's response has been severely criticised by fellow academics in the past for bogus low quality modelling. The Imperial College paper has basic errors and flawed assumptions that are obvious even to untrained laymen.
I think you're wildly over-estimating how much statistical and logical training most academics get. This is one of the basic underlying problems that the replication crisis has exposed: an unending flood of academic papers, especially those doing modelling, that fall apart when examined by people with statistical and mathematical training.
You're right that a 4 week bootcamp won't make someone a flawless handling of data. But it might still be a more rigorous form of training than epidemiologists get.
I agree with points from the parent comment, but more so from you. It's good to have experts who've rigorously studied historical practices and their outcomes; we can learn many things about which practices work best, and what mathematical relationships they reveal. But history is not an accurate predictor of the future, and theory is only a stepping stone for better testing (such as Great Britain's theory for herd immunity, which was a ground breaking theoretical approach to dealing with a pandemic).
I think the most effective approach, which adds to your argument, is ensemble modelling from numerous, independent researchers from many fields. This is akin to how many people guessing at the number of gumballs in a container at the fair will average to the correct amount, while any single guess will not. There is a Freakonomics episode, Superpredictors, which demonstrates the use of this statistical approach for anticipating the outcome of voting in foreign politics.
I think the most effective approach, which adds to your argument, is ensemble modelling from numerous, independent researchers from many fields. This is akin to how many people guessing at the number of gumballs in a container at the fair will average to the correct amount, while any single guess will not. There is a Freakonomics episode, Superpredictors, which demonstrates the use of this statistical approach for anticipating the outcome of voting in foreign politics.
While I agree that mocking people for self-learning is atrocious, I don't understand why in the same breath, you felt compelled to mock a university based education.
Some people thrive in the bootcamp scene, while others enjoy the structure of a university. Both can be great tools for self-learning (I am thinking specifically of the resources a large university can provide, like media and electronics labs for creative projects that involve expensive cameras, 3D printers, etc), and both can be misused. University based education does not have to be ancient.
Some people thrive in the bootcamp scene, while others enjoy the structure of a university. Both can be great tools for self-learning (I am thinking specifically of the resources a large university can provide, like media and electronics labs for creative projects that involve expensive cameras, 3D printers, etc), and both can be misused. University based education does not have to be ancient.
Self learning would imply that the person is teaching themselves, not that the person is attending a coding boot camp.
People that claim to be data scientists after attending a coding boot camp must be clowns. I’ve never come across such a person. All the data scientists I know are smart self taught people, not people who went to a boot camp.
People that claim to be data scientists after attending a coding boot camp must be clowns. I’ve never come across such a person. All the data scientists I know are smart self taught people, not people who went to a boot camp.
It's possible that all the data scientists you know are not a representative sample.
I've met smart data scientists and engineers who've gone to a bootcamp. People who are good and bad at their jobs come from a huge variety of backgrounds, and it's not helpful to look down on folks whose background is different than your own.
I've met smart data scientists and engineers who've gone to a bootcamp. People who are good and bad at their jobs come from a huge variety of backgrounds, and it's not helpful to look down on folks whose background is different than your own.
I am not looking down on folks different from myself. Also, I am not a data scientist. Regardless, what you are claiming is not possible.
As I am sure you know, data science is a specialized field, one which draws upon a combination of statistics and computer science practices. You may be able to get some of the computer science practices down in a boot camp, but there's no way you're getting a boot camp to fill in the Bachelor's in statistics you need to be a good data scientist. Years of hard work and sincere dedication, however, I am sure can get you there.
As I am sure you know, data science is a specialized field, one which draws upon a combination of statistics and computer science practices. You may be able to get some of the computer science practices down in a boot camp, but there's no way you're getting a boot camp to fill in the Bachelor's in statistics you need to be a good data scientist. Years of hard work and sincere dedication, however, I am sure can get you there.
The issue with is the word "scientist".
In academia, science is the pursuit of postulating a falsifiable theory, then gathering facts to verify that theory, and then going through a process of intensive peer review that either confirms, dispels or amends those findings.
The formality of the academic process is a necessity. The value of scientific findings entirely depends on the trustworthiness of the research. That is, how were the results obtained, which line of thinking was followed, did the research exclude crucial biases, etc.?
The importance here is that academic research is used as a pillar to produce products and services we all use in daily life. For instance, if you need a hip replacement, you want to be sure that prosthetic was designed based on rigorous scientific findings and studies that can vouch for safety and comfort.
The difference then with "data scientist" is that they often they don't apply the same rigorous research practices. It's easy to pick a dataset and bang off visualizations; it's a different story to actually come up with relevant questions, assert the quality of the data at hand and publish your findings towards a community of domain experts who are actually able to review your findings.
One needs in-depth domain knowledge to do that. Good data scientists will understand this limitation. They often work in a specific domain in a supporting capacity: bringing technical skills and capabilities to domain experts that don't have those skills.
Then there are those who purport to practice data science while grokking datasets, creating visualisations and cobbling a blogpost together at the end of the day. That's when you need to be really wary of what they publish, even if the bigger picture contains truthiness.
Hence why I have extremely mixed feelings about what https://medium.com/@tomaspueyo is doing.
To be sure, the core points of what he's telling are in line with what domain experts are telling us. But the extreme number juggling is quite mind bending. Moreover, the man is not a domain expert. He's an entrepreneur who happens to know how to write viral blogposts such as "how to deliver your funny speech" and "How to become the best in the world at something". What he does is anything but scientific. And so, even though he's making a heartfelt plea heard by many, one should be careful to not take the precise details in his pieces at face value.
At the moment, we all are victims of our own confirmation bias. Each day yields another data point, and given our desperate state, we want to see trends that confirm improvement, a probability that one will survive this, low mortality and so on. The reality is that we only have so few datapoints and it's still far too soon to make conclusive assertions about how this will pan out for the world at large and you in particular.
In academia, science is the pursuit of postulating a falsifiable theory, then gathering facts to verify that theory, and then going through a process of intensive peer review that either confirms, dispels or amends those findings.
The formality of the academic process is a necessity. The value of scientific findings entirely depends on the trustworthiness of the research. That is, how were the results obtained, which line of thinking was followed, did the research exclude crucial biases, etc.?
The importance here is that academic research is used as a pillar to produce products and services we all use in daily life. For instance, if you need a hip replacement, you want to be sure that prosthetic was designed based on rigorous scientific findings and studies that can vouch for safety and comfort.
The difference then with "data scientist" is that they often they don't apply the same rigorous research practices. It's easy to pick a dataset and bang off visualizations; it's a different story to actually come up with relevant questions, assert the quality of the data at hand and publish your findings towards a community of domain experts who are actually able to review your findings.
One needs in-depth domain knowledge to do that. Good data scientists will understand this limitation. They often work in a specific domain in a supporting capacity: bringing technical skills and capabilities to domain experts that don't have those skills.
Then there are those who purport to practice data science while grokking datasets, creating visualisations and cobbling a blogpost together at the end of the day. That's when you need to be really wary of what they publish, even if the bigger picture contains truthiness.
Hence why I have extremely mixed feelings about what https://medium.com/@tomaspueyo is doing.
To be sure, the core points of what he's telling are in line with what domain experts are telling us. But the extreme number juggling is quite mind bending. Moreover, the man is not a domain expert. He's an entrepreneur who happens to know how to write viral blogposts such as "how to deliver your funny speech" and "How to become the best in the world at something". What he does is anything but scientific. And so, even though he's making a heartfelt plea heard by many, one should be careful to not take the precise details in his pieces at face value.
At the moment, we all are victims of our own confirmation bias. Each day yields another data point, and given our desperate state, we want to see trends that confirm improvement, a probability that one will survive this, low mortality and so on. The reality is that we only have so few datapoints and it's still far too soon to make conclusive assertions about how this will pan out for the world at large and you in particular.
Well as a statistician, I hate to break it to you, but there is a massive reproduction crisis in science because heretofore the "rigourous research practices" has often times been more a veneer of rigor rather than actual science. The amount of published research (often in esteemed journals) that has been found to be unreproducible and based on faulty methods is aburdly high. The modern scientific method has arguably been as or more effective at building a tower that shields scientists and academics from criticism from work of questionable validity.
I'm not saying random data science blogs aren't often wrong. But you've probably been burned just as often by publish science and simply haven't realized it. And at least the data science blogs aren't behind expensive paywalls, aren't couched in meaningless vernacular, and present the code/data for reproducing their results, none of which can be said for a lot of science these days.
I'm not saying random data science blogs aren't often wrong. But you've probably been burned just as often by publish science and simply haven't realized it. And at least the data science blogs aren't behind expensive paywalls, aren't couched in meaningless vernacular, and present the code/data for reproducing their results, none of which can be said for a lot of science these days.
Well, being professionally active in academia, I'm well aware of this issue and that particular debate. And you're right.
Open science and open access are important movements as to the relationship between publishers and academic research.
However, that debate doesn't negate my point as to citizen or data science.
For all the discussion about paywalls and the use of vernacular, the same timeless critical considerations need applying: who wrote the article? what is their background? are they a domain expert? where they did get their sources? how did they come to conclusions? is their method sound? are they asking relevant questions? etc. etc.
And I don't always see that happening online. On the contrary. The fluidity and the speed at which information flows online seems to be a justification to give in and accept what's being said at face value. The past few years should have made it clear that such indulgence can lead to dire consequences.
With powerful tools comes responsibility. Sure, it's great to see people apply free and open source tools to come to a new understanding of observations. But that's only half of the story: you still have to apply critical thinking to those conclusions. That's not something throwing more code or technical skills can achieve.
Having a proper, critical debate takes time and experience.
Open science and open access are important movements as to the relationship between publishers and academic research.
However, that debate doesn't negate my point as to citizen or data science.
For all the discussion about paywalls and the use of vernacular, the same timeless critical considerations need applying: who wrote the article? what is their background? are they a domain expert? where they did get their sources? how did they come to conclusions? is their method sound? are they asking relevant questions? etc. etc.
And I don't always see that happening online. On the contrary. The fluidity and the speed at which information flows online seems to be a justification to give in and accept what's being said at face value. The past few years should have made it clear that such indulgence can lead to dire consequences.
With powerful tools comes responsibility. Sure, it's great to see people apply free and open source tools to come to a new understanding of observations. But that's only half of the story: you still have to apply critical thinking to those conclusions. That's not something throwing more code or technical skills can achieve.
Having a proper, critical debate takes time and experience.
> who wrote the article?
This seems to be an improvement over academia imo. The person writing the article should be immaterial. The same paper written by an unheard of researcher should be treated no differently than the same paper written by a well-known, tenured researcher at a prestigious institution. Unfortunately in academia, who you are and who you know is often as big a deal as what you do and what you know.
> What is their background? are they a domain expert?
Again, the methodology should stand alone. Being a "domain expert" or having a particular background is only relevant if the readers of the article are incapable of judging the results on their own merits, and have to instead rely appeals to authority. But appeals to authority introduce their own problems, including a moat that protects the status quo and the ingroup from potentially important new ideas and newcomers. And its hardly scientific.
> where they did get their sources? how did they come to conclusions? is their method sound? are they asking relevant questions? etc. etc.
These need not be restricted to academic work. If a blog fails to properly cite sources, you can move on. Plenty of academic papers are built on flimsy premises, poor methodology, asking incorrect questions, etc. etc. There is nothing differentiating the ability to assess the quality of something written in a blog vs something written in an academic journal here.
> And I don't always see that happening online. On the contrary. The fluidity and the speed at which information flows online seems to be a justification to give in and accept what's being said at face value. The past few years should have made it clear that such indulgence can lead to dire consequences.
And yet, as noted, the more conservative approach of academia has resulted in a huge amount of unreproducible "science". And now, in a time of crisis, the stringent model is being set aside in favor of openly available preprints shared online, without peer review, via social media. How can the traditional model be considered useful when, in times without urgency, the validity of its results are highly questionable, and in times of urgency, its thrown aside for the sake of actual progress? It seems to me the modern academic method is largely a facade, something closer to an ornate religious ritual that has been divorced of its actual intentions.
> But that's only half of the story: you still have to apply critical thinking to those conclusions. That's not something throwing more code or technical skills can achieve.
This assumes that those coding or utilizing technical skills are doing so without critical thinking. Perhaps this is true some of the time, but I don't see the situation being any different in academia.
> Having a proper, critical debate takes time and experience.
What constitutes a "proper, critical debate" is subjective. And from my position, it would appear that the advent of the internet and freedom of information is making the current academic model unsustainable and obsolete. And so academia has an entrenched interest in defining "proper, critical debate" in a way that protects their livelihood.
I'm not saying there isn't plenty of poorly written science and analysis being done on blogs out there. I'm simply saying I don't think traditional academia and science is much different.
This seems to be an improvement over academia imo. The person writing the article should be immaterial. The same paper written by an unheard of researcher should be treated no differently than the same paper written by a well-known, tenured researcher at a prestigious institution. Unfortunately in academia, who you are and who you know is often as big a deal as what you do and what you know.
> What is their background? are they a domain expert?
Again, the methodology should stand alone. Being a "domain expert" or having a particular background is only relevant if the readers of the article are incapable of judging the results on their own merits, and have to instead rely appeals to authority. But appeals to authority introduce their own problems, including a moat that protects the status quo and the ingroup from potentially important new ideas and newcomers. And its hardly scientific.
> where they did get their sources? how did they come to conclusions? is their method sound? are they asking relevant questions? etc. etc.
These need not be restricted to academic work. If a blog fails to properly cite sources, you can move on. Plenty of academic papers are built on flimsy premises, poor methodology, asking incorrect questions, etc. etc. There is nothing differentiating the ability to assess the quality of something written in a blog vs something written in an academic journal here.
> And I don't always see that happening online. On the contrary. The fluidity and the speed at which information flows online seems to be a justification to give in and accept what's being said at face value. The past few years should have made it clear that such indulgence can lead to dire consequences.
And yet, as noted, the more conservative approach of academia has resulted in a huge amount of unreproducible "science". And now, in a time of crisis, the stringent model is being set aside in favor of openly available preprints shared online, without peer review, via social media. How can the traditional model be considered useful when, in times without urgency, the validity of its results are highly questionable, and in times of urgency, its thrown aside for the sake of actual progress? It seems to me the modern academic method is largely a facade, something closer to an ornate religious ritual that has been divorced of its actual intentions.
> But that's only half of the story: you still have to apply critical thinking to those conclusions. That's not something throwing more code or technical skills can achieve.
This assumes that those coding or utilizing technical skills are doing so without critical thinking. Perhaps this is true some of the time, but I don't see the situation being any different in academia.
> Having a proper, critical debate takes time and experience.
What constitutes a "proper, critical debate" is subjective. And from my position, it would appear that the advent of the internet and freedom of information is making the current academic model unsustainable and obsolete. And so academia has an entrenched interest in defining "proper, critical debate" in a way that protects their livelihood.
I'm not saying there isn't plenty of poorly written science and analysis being done on blogs out there. I'm simply saying I don't think traditional academia and science is much different.
What Tomas Pueyo did is create a massively viral piece of content that, regardless of accuracy, encouraged the right actions. In doing so he saved at least thousands of lives. So forgive me for not caring how robust his analyses are.
Saving lives doesn't absolve anyone from critical consideration about what they are publishing exactly.
For all intents and purposes, there are authors who write similar massive viral pieces that may - and likely will - end up killing thousands of people inadvertently.
As I said, the core message is absolutely right, but his method - the way he packs his message with solid looking graphs and number juggling - is questionable. Sure, it gets the job done; whereas many others fail to push the very same message. But it's still a questionable tactic of convincing people.
Does it matter? Perhaps not. At the end of the day, he saved lives. Morality is a luxury presently. And yet, dismissing critical considerations outright equally opens the door to unintended consequences if we turn it into a habit each time someone is purported to have saved lives.
For all intents and purposes, there are authors who write similar massive viral pieces that may - and likely will - end up killing thousands of people inadvertently.
As I said, the core message is absolutely right, but his method - the way he packs his message with solid looking graphs and number juggling - is questionable. Sure, it gets the job done; whereas many others fail to push the very same message. But it's still a questionable tactic of convincing people.
Does it matter? Perhaps not. At the end of the day, he saved lives. Morality is a luxury presently. And yet, dismissing critical considerations outright equally opens the door to unintended consequences if we turn it into a habit each time someone is purported to have saved lives.
Why is bootcamp bad? How'bout schools?
Self motivation is much harder than you think, and a little nudge from a bootcamp may be what most people need before they become 'self motivated'.
Self motivation is much harder than you think, and a little nudge from a bootcamp may be what most people need before they become 'self motivated'.
Any amount of coding ... involves self learning regardless of what class(es) you took.
Lets also not give machine guns to monkeys and let them loose in crowds.
There is a time and place for naive, exploratory, or just plain wrong-headed exposition. Making plausible looking but wrong "scientific" prognostications in the middle of a global emergency isn't one of them.
There is a time and place for naive, exploratory, or just plain wrong-headed exposition. Making plausible looking but wrong "scientific" prognostications in the middle of a global emergency isn't one of them.
The irony might be the massive human scale effort the NYT is putting in to gather and vet the data. The most interesting self learning side effort for this, might not be analyzing the data, but trying to automate the data collection from public sites checked against the nyt data.
I don't think people care about the source of learning. We care about the people who fall into the Dunning-Kruger effect and think their self learning puts them above people who dedicated a large portion of their lives to learning the same subject. An important part of self learning is being aware of your own level of knowledge. So feel free to play around with this dataset. Feel free to post to Medium what you find as long as you are clear it is a learning exercise. You don't want to actively mislead or be used as a prop for others to mislead especially during this time of crisis.
> Feel free to post to Medium what you find as long as you are clear it is a learning exercise. You don't want to actively mislead or be used as a prop for others to mislead especially during this time of crisis.
Some people can profit from this. Hence, the incentive.
Some people can profit from this. Hence, the incentive.
"Oh boy, I can hear the sophomore software engineering studio students firing up their word processors from here" works just as well.
I don't think the critique was of self-learning or bootcamps. It was a critique of a certain sort of cringe-worthy self-promotion.
And not just cringe-worthy. Spreading information based upon a cursory analysis of some CSV files without any training or experience in epidemiology, public health, public communications, etc. seems... irresponsible.
Also, off-topic, but most of my peers at university taught themselves how to program years before starting college -- typically in middle/early high school. And I mean actually taught themselves, from books and zine tutorials, not 'attended a formal course of instruction that was offered by a venture-backed firm instead of a formal course of instruction at a traditional university'. I guess times have changed, but the characterization of university students in CS as 'not self-taught programmers' is definitely the opposite of my experience. You came into the CS degree knowing how to program and learned how to do CS. And the characterization of "anything not university" as "self-taught" -- even formal courses of instruction that cost five figures -- is even more strange.
I don't think the critique was of self-learning or bootcamps. It was a critique of a certain sort of cringe-worthy self-promotion.
And not just cringe-worthy. Spreading information based upon a cursory analysis of some CSV files without any training or experience in epidemiology, public health, public communications, etc. seems... irresponsible.
Also, off-topic, but most of my peers at university taught themselves how to program years before starting college -- typically in middle/early high school. And I mean actually taught themselves, from books and zine tutorials, not 'attended a formal course of instruction that was offered by a venture-backed firm instead of a formal course of instruction at a traditional university'. I guess times have changed, but the characterization of university students in CS as 'not self-taught programmers' is definitely the opposite of my experience. You came into the CS degree knowing how to program and learned how to do CS. And the characterization of "anything not university" as "self-taught" -- even formal courses of instruction that cost five figures -- is even more strange.
That just doesn't seem like a fair standard. Why should only experts in a relevant field talk about the coronavirus? Nobody tries to stop people from sharing hospital stories from their friend's cousin's boyfriend who's a nurse.
Oh, I think it's completely fine to play with the data and talk about it with friends.
But SEOd medium posts have different reach potential than a facebook comment/phone call with friends/family. That's kind of the whole point of them.
But SEOd medium posts have different reach potential than a facebook comment/phone call with friends/family. That's kind of the whole point of them.
I get what you're saying, but I don't consider the two things separate magisteria. I don't think there's anything generally wrong with people trying to spread honest ideas they've had, even if they aren't experts.
For something like Covid, I'd question how much time should be spent on "trying to spread" vs the amount of time spent on refining the idea.
One thing about experts is that they’ve usually made it through the Dunning-Kruger “confidence pit” and come out the other side. They’re often cautious about making very strident remarks, and when they are really confident in a conclusion, it’s usually because it has a very high probability of being correct. When someone new to a field comes to a contrarian conclusion, they’re often willing to show more confidence in their conclusion because of the Dunning-Kruger effect. This makes it hard to take non-experts at their word, since they’re often unaware of how likely they are to be wrong.
Now, if a total non-expert had come out of nowhere with an analysis that contradicted the public health establishment, definitively showed that the danger was massively overblown, and was right, that would be one thing. But this medium post was incorrect in its conclusion, quite confident in its conclusion, and advised courses of action that would put many people in danger (such as reopening schools).
Now, if a total non-expert had come out of nowhere with an analysis that contradicted the public health establishment, definitively showed that the danger was massively overblown, and was right, that would be one thing. But this medium post was incorrect in its conclusion, quite confident in its conclusion, and advised courses of action that would put many people in danger (such as reopening schools).
I'm not sure which medium post you're referring to here. I'm sure people do sometimes write incorrect advice on Medium that would be dangerous if followed, but experts can do the same thing. Remember when the WHO said not to ban travel?
I'm also not sure that distinguishes the case I mentioned. It might also cause problems to assume some particular anecdote is true and representative.
I'm also not sure that distinguishes the case I mentioned. It might also cause problems to assume some particular anecdote is true and representative.
That's a very romanticised view of expertise that doesn't match what is seen in the real world. If your view were true the replication crisis would not be happening, but it's barely even got started.
Self promotion is a skill. If it makes you cringe you need to get better at it.
Not all self promotion is good self promotion. I've nuked candidates because their online portfolios demonstrate bad technical judgement, questionable professional ethics, a fundamental misunderstanding of core ideas, etc.
Put more simply: a portfolio is a demonstration of your work. Anything you put out in public with your name on it is part of your portfolio. Obviously, don't put bad work in your public portfolio.
There's a reason you see very few serious data scientists publishing hot takes on their medium blogs -- it's a serious threat to your professional integrity and brand if you get it wrong. The only people I know & respect who are writing publicly about this topic have SME collaborators. (Of course, we're all playing with the data and talking about it in private with friends/coworkers over coffee.)
Self promotion specifically by spreading your amateur take on a health crisis seems a bit... immoral? Or at least the sort of thing that makes me question the candidate's judgement.
If you want to write publicly and in a formal way on this topic, get a subject matter expert to serve as a co-author. Anything less is more risky than it's worth, even from a purely selfish perspective.
Put more simply: a portfolio is a demonstration of your work. Anything you put out in public with your name on it is part of your portfolio. Obviously, don't put bad work in your public portfolio.
There's a reason you see very few serious data scientists publishing hot takes on their medium blogs -- it's a serious threat to your professional integrity and brand if you get it wrong. The only people I know & respect who are writing publicly about this topic have SME collaborators. (Of course, we're all playing with the data and talking about it in private with friends/coworkers over coffee.)
Self promotion specifically by spreading your amateur take on a health crisis seems a bit... immoral? Or at least the sort of thing that makes me question the candidate's judgement.
If you want to write publicly and in a formal way on this topic, get a subject matter expert to serve as a co-author. Anything less is more risky than it's worth, even from a purely selfish perspective.
Self promotion with ??? analysis about public health crisis should make EVERYONE cringe.
Love your layers of hate-keeping. Not only did they learn before uni, they actually learned before uni, none of that fake bs!
Never mind that most of the people learning before uni did so from opportunity - they often had an engineer in the family or lived in a well-to-do area where such skills were apparent. (This magnified even further the older you are).
FWIW, I studied CS at a great cs uni (cal), and most of my peers were not self-taught. They still ended up at the same companies in the same positions as those who did ️.
Never mind that most of the people learning before uni did so from opportunity - they often had an engineer in the family or lived in a well-to-do area where such skills were apparent. (This magnified even further the older you are).
FWIW, I studied CS at a great cs uni (cal), and most of my peers were not self-taught. They still ended up at the same companies in the same positions as those who did ️.
> I studied CS at a great cs uni
Do you feel that your time there was wasted, and that you would have been better off attending a more focused bootcamp?
Do you feel that your time there was wasted, and that you would have been better off attending a more focused bootcamp?
Not at all. My twin brother went that route, and we’re both at FAANG. I also don’t feel any different for not having done much cs before uni.
Sadly all of this data assumes there are approximately as many beds as needed, and won't be very helpful `x` weeks from now because once we hit capacity, the situation is bound to look very different.
I don't see enough people making this point: the young will be very vulnerable if left untreated, and you can't treat e.g. the entire state of NY with 3,800 ICU beds plus however many the wonderful people in the Army Corps of Engineer can put together in a pinch.
It's not like young people are going to recover at home from their nasty case of pneumonia. With no ventilators and no medical staff, the mortality rate will markedly increase.
I don't see enough people making this point: the young will be very vulnerable if left untreated, and you can't treat e.g. the entire state of NY with 3,800 ICU beds plus however many the wonderful people in the Army Corps of Engineer can put together in a pinch.
It's not like young people are going to recover at home from their nasty case of pneumonia. With no ventilators and no medical staff, the mortality rate will markedly increase.
> you can't treat e.g. the entire state of NY with 3,800 ICU beds plus however many the wonderful people in the Army Corps of Engineer can put together in a pinch.
Can't you? Cuomo seems pretty optimistic that once they get their 30,000 ventilators they'll be able to handle the peak.
Can't you? Cuomo seems pretty optimistic that once they get their 30,000 ventilators they'll be able to handle the peak.
The over capacity modeling is incredibly difficult to do. Please stop trying to spread additional gloom and doom about a part of this future most people have no control over.
There are already novel ideas to address that issue being put into use and capacity is ramping up rapidly.
Modeling the future is incredibly difficult. If you think it is so easy, feel free to go become a billionaire in the financial markets and donate the money to help with the cause.
Spread fear about uncontrollable and unpredictable future events is wildly irresponsible.
There are already novel ideas to address that issue being put into use and capacity is ramping up rapidly.
Modeling the future is incredibly difficult. If you think it is so easy, feel free to go become a billionaire in the financial markets and donate the money to help with the cause.
Spread fear about uncontrollable and unpredictable future events is wildly irresponsible.
Great ad hominem, but I can't resist responding even though you're basically just repeating the same (wrong) argument multiple times.
> The over capacity modeling is incredibly difficult to do.
I never stated otherwise. And I'm not asking anyone to model anything. I'm just saying we'll be over capacity at a near point in the future. That seems like a pretty defensible extrapolation, but would love to hear how that's not the case.
> Please stop trying to spread additional gloom and doom about a part of this future most people have no control over.
Why? This is the world we live in currently. What you call "gloom and doom" others would call being cautious.
> There are already novel ideas to address that issue being put into use and capacity is ramping up rapidly.
Ideas. Let's hope they come to fruition. In the meantime, let's not kid ourselves about the risk to the non-elderly.
> Modeling the future is incredibly difficult.
Again irrelevant. I never claimed otherwise.
> If you think it is so easy
I don't think it's easy
> Spread fear about uncontrollable and unpredictable future events is wildly irresponsible.
I'm not "spreading fear". I'm debating an issue with most analyses currently being done in that they don't apply to the overcapacity scenario. And I disagree that's an irresponsible argument to make. Indeed, I can't see how turning a deaf ear to this most reasonable extrapolation that we'll soon be over capacity isn't the most irresponsible of the two.
> The over capacity modeling is incredibly difficult to do.
I never stated otherwise. And I'm not asking anyone to model anything. I'm just saying we'll be over capacity at a near point in the future. That seems like a pretty defensible extrapolation, but would love to hear how that's not the case.
> Please stop trying to spread additional gloom and doom about a part of this future most people have no control over.
Why? This is the world we live in currently. What you call "gloom and doom" others would call being cautious.
> There are already novel ideas to address that issue being put into use and capacity is ramping up rapidly.
Ideas. Let's hope they come to fruition. In the meantime, let's not kid ourselves about the risk to the non-elderly.
> Modeling the future is incredibly difficult.
Again irrelevant. I never claimed otherwise.
> If you think it is so easy
I don't think it's easy
> Spread fear about uncontrollable and unpredictable future events is wildly irresponsible.
I'm not "spreading fear". I'm debating an issue with most analyses currently being done in that they don't apply to the overcapacity scenario. And I disagree that's an irresponsible argument to make. Indeed, I can't see how turning a deaf ear to this most reasonable extrapolation that we'll soon be over capacity isn't the most irresponsible of the two.
> Spread fear about uncontrollable and unpredictable future events is wildly irresponsible.
I'm not sure about that. Back in January, it was surely not irresponsible to get people worried about a pandemic. China is often criticized for suppressing online fear-mongering. Climate change activists are praised for it. Fear motivates people. It might not be the most rational motivation but neither is passively believing everything will continue as normal. This idea that the world is full of gullible idiots and that "us" smart people have to be careful what we say in case they act on it is arrogant. Facts online are often self-correcting, as you can see right here with people immediately challenging his claim.
I'm not sure about that. Back in January, it was surely not irresponsible to get people worried about a pandemic. China is often criticized for suppressing online fear-mongering. Climate change activists are praised for it. Fear motivates people. It might not be the most rational motivation but neither is passively believing everything will continue as normal. This idea that the world is full of gullible idiots and that "us" smart people have to be careful what we say in case they act on it is arrogant. Facts online are often self-correcting, as you can see right here with people immediately challenging his claim.
> I don't see enough people making this point: the young will be very vulnerable if left untreated
Why are you pointing out "the young" here?
Why are you pointing out "the young" here?
Because the prevailing narrative is that this disease is mainly dangerous to the elderly.
It is primarily dangerous to the elderly. That is what the actual data shows.
Some people are pushing a counter-narrative that youngs aren't invincible. Which is also true. 0.1% death rates doesn't mean zero deaths. A higher number will be hospitalized, and as hospitals become overwhelmed that means (a) quality of care will be degraded and death rates will increase somewhat, and (b) any youngs in a bed comes at the direct expense of an elderly in a bed.
But if you are claiming that this disease is not mainly dangerous to the elderly, no, that is false, the deaths and hospitalizations are hugely weighted to the elderly and are quite low for young cohorts.
Yes, it is important that everyone minimize contacts as much as possible. For some reason people love to really harp on this point with the youngs. I didn't go to florida for spring break, it was my boomer parents who did (or rather, were planning on going this summer and complained nonstop that Florida state parks closed and cut them off). Churches are still open in many states spreading disease. Boomers are still going to Home Depot and buying their home improvement goods. A lot of boomers aren't taking this seriously either but nobody chooses to bring that up.
There are lots of changes that need to be made by everyone, but younger people are one of the primary targets of casual/unthinking ageism in this country. Avocado toast and all that.
Focusing the blame on this onto the young is no more correct, helpful, or productive than focusing the blame on the Chinese, and I suspect is coming from most of the same media sources.
Some people are pushing a counter-narrative that youngs aren't invincible. Which is also true. 0.1% death rates doesn't mean zero deaths. A higher number will be hospitalized, and as hospitals become overwhelmed that means (a) quality of care will be degraded and death rates will increase somewhat, and (b) any youngs in a bed comes at the direct expense of an elderly in a bed.
But if you are claiming that this disease is not mainly dangerous to the elderly, no, that is false, the deaths and hospitalizations are hugely weighted to the elderly and are quite low for young cohorts.
Yes, it is important that everyone minimize contacts as much as possible. For some reason people love to really harp on this point with the youngs. I didn't go to florida for spring break, it was my boomer parents who did (or rather, were planning on going this summer and complained nonstop that Florida state parks closed and cut them off). Churches are still open in many states spreading disease. Boomers are still going to Home Depot and buying their home improvement goods. A lot of boomers aren't taking this seriously either but nobody chooses to bring that up.
There are lots of changes that need to be made by everyone, but younger people are one of the primary targets of casual/unthinking ageism in this country. Avocado toast and all that.
Focusing the blame on this onto the young is no more correct, helpful, or productive than focusing the blame on the Chinese, and I suspect is coming from most of the same media sources.
> It is primarily dangerous to the elderly. That is what the actual data shows.
Right, because the mortality rate for people under intensive care is skewed towards the elderly / people with underlying health conditions because they can't weather the ICU storm.
My point is that data is irrelevant for the scenario in which people go without intensive care. We have no idea how deadly COVID-19 will be then (or at least I haven't seen any data on those cases)
Right, because the mortality rate for people under intensive care is skewed towards the elderly / people with underlying health conditions because they can't weather the ICU storm.
My point is that data is irrelevant for the scenario in which people go without intensive care. We have no idea how deadly COVID-19 will be then (or at least I haven't seen any data on those cases)
> Boomers are still going to Home Depot and buying their home improvement goods.
It is worth pointing out that home-improvement stores were specifically exempt from the order to close businesses in several European countries. This is probably partly because those stores sell gardening products, and villagers need to be able to plant their vegetables now. However, the expectation on the part of public health authorities is presumably that that these stores remaining open, like supermarkets, will still not prevent the infection curve from flattening over the lockdown period.
It is worth pointing out that home-improvement stores were specifically exempt from the order to close businesses in several European countries. This is probably partly because those stores sell gardening products, and villagers need to be able to plant their vegetables now. However, the expectation on the part of public health authorities is presumably that that these stores remaining open, like supermarkets, will still not prevent the infection curve from flattening over the lockdown period.
When I see datasets like this, I just think of how 95% of my work as a DS has been just getting to that cleaned table at the end. After that the modeling is straightforward.
“Math is boring. When am I ever going to use it?”
The data can be used by a lot of people for a lot of learning.
My project: I’d like to watch the growth state by state on a daily basis.
The data can be used by a lot of people for a lot of learning.
My project: I’d like to watch the growth state by state on a daily basis.
If you want an example of how to do that good, I find this the best:
https://www.ft.com/coronavirus-latest
as a simpler start, see the logarithmic chart here:
https://www.worldometers.info/coronavirus/country/us/
Note how easy is to estimate the time frame in which the next factor of 10 could be reached with the growth that continues.
Compare with the logarithmic chart of China:
https://www.worldometers.info/coronavirus/country/china/
Search for the date since when they had their measures applied in Wuhan. Then follow how long it took for them to be visible in the chart. Then mark on the US chart when the measures started. Compare. Also mark the day of the following quote: "And we’re prepared, and we’re doing a great job with it. And it will go away. Just stay calm. It will go away."
One can learn really a lot just from the data.
https://www.ft.com/coronavirus-latest
as a simpler start, see the logarithmic chart here:
https://www.worldometers.info/coronavirus/country/us/
Note how easy is to estimate the time frame in which the next factor of 10 could be reached with the growth that continues.
Compare with the logarithmic chart of China:
https://www.worldometers.info/coronavirus/country/china/
Search for the date since when they had their measures applied in Wuhan. Then follow how long it took for them to be visible in the chart. Then mark on the US chart when the measures started. Compare. Also mark the day of the following quote: "And we’re prepared, and we’re doing a great job with it. And it will go away. Just stay calm. It will go away."
One can learn really a lot just from the data.
I think its amazing that people are passionate, but the humility about not understanding something in its entirety is shrinking day by day. Today everyone wants to be the "explainer" or the "let me tell you how its done" person without actually having any hands-on experience or deep knowledge in the domain. The oldskool litmus test was "What have you done in this field that makes your opinion worth something?" and it is clearly no longer applicable in today's world.
Here’s a quickie animated bubble map, pulling in this new NYT data from GitHub: https://observablehq.com/@jashkenas/united-states-coronaviru...
... might be a useful starting point if you want to fork off your own visualizations and analyses.
... might be a useful starting point if you want to fork off your own visualizations and analyses.
At first I thought your animation wasn't working. Then it got to the last few days.
Wow.
Wow.
Yeah. Wow indeed. The dataset actually goes all the way back to the end of January, but I'm trimming it to start at the beginning of March, because the early days are pretty much invisible at the current scale.
You can change the value of `startDate` in that notebook if you'd like to look back further in time.
You can change the value of `startDate` in that notebook if you'd like to look back further in time.
You could make the animation slow down exponentially, to make the growth look linear. Basically turn it into a log plot. Misleading if you don't know what you're looking at though.
Anybody know about data indicating age? It's always "confirmed", "dead", "recovered" but no info about age of patients.
In addition to that, I want to know the number of hospitalizations or deaths in those age groups. Even better would be to know the percentage of those hospitalized/dead that had underlying medical conditions.
This has that data:
https://www.cdc.gov/mmwr/volumes/69/wr/mm6912e2.htm
My state publishes its statistics hourly. I looked at it just this morning, and was surprised to see that in the age breakdowns that there are more confirmed cases in people 20-40 than in 60-80.
I am not surprised. People in the 20-40 range are more likely to be out in public exposing themselves to the virus. The elderly are by default socially distanced in this country. There has been no evidence I've seen that age affects likelihood of infection, but it does affect severity.
In countries like Italy where very few young people have tested positive, it's due to test triage. You aren't going to spend tests on asymptomatic or mildly symptomatic 25yos.
In countries like Italy where very few young people have tested positive, it's due to test triage. You aren't going to spend tests on asymptomatic or mildly symptomatic 25yos.
The absolute numbers aren't that interesting; you need those numbers as a percentage of the total people in that demographic.
Is that still true when you adjust for the relative size of those groups? 20-40 is a larger group of people than 60-80.
It's hard to say, because I don't have the time to do the math right now. But my state is generally considered to be retirement-friendly.
Florida?
Do you have a link to that data? Thanks.
NYC publishes this kind of data ("Daily Syndromic and Case Data Update"): https://www1.nyc.gov/site/doh/covid/covid-19-main.page
This website appears to have more fine-grained statistics for the New York City area (ironically): https://accelerator.weather.com/bi/?perspective=dashboard&pa...
I don't understand why these lists make it so hard to find the USA. Every country is written out including the United Kingdom - but the USA is "US." Sometimes we're first because of ethnocentrism, sometimes last, but PLEASE BE CONSISTENT!
Also, thanks for this link - rant aside, it's the best dataset I've seen.
Also, thanks for this link - rant aside, it's the best dataset I've seen.
This is phenomenal. I've been scraping the data from primary sources for just Washington state for the past week [0], in order to make this chart which I hacked together last weekend [1].
[0]: https://github.com/jake-low/covid-19-wa-data
[1]: https://observablehq.com/@jake-low/covid-19-in-washington-st...
Doing this for just one state was a pretty substantial effort. I imagine there are multiple people at the Times who are spending several hours a day reviewing and cleaning scraped data (seems every couple of days some formatting change breaks your scripts, or a source publishes data that later needs to be retracted).
The Times dataset appears to contain per-county case and death observations in a time series, going all the way back to the first confirmed U.S. case in January in Snohomish County, WA. This makes it by far the most comprehensive time series dataset of U.S. COVID-19 cases publicly available.
Some people in this thread linked to the Johns Hopkins CSSE dataset; I've looked at this data but it doesn't go back very far in time for the U.S., and the tables are published as daily summaries with differing table schemas which makes them hard to use out of the box. For some days earlier in March, "sublocations" aren't even structured (for example the same column contains, "Boston, MA" and "Los Angeles County", making it very hard to use). No disrespect to the team behind the JHU dataset; it attempts to cover the whole world since the outbreak began which is an incredible and difficult goal. But for mapping and studying the outbreak in the U.S., the Times dataset will likely be the best choice right now.
Huge kudos to the New York Times team for making this data freely available.
[0]: https://github.com/jake-low/covid-19-wa-data
[1]: https://observablehq.com/@jake-low/covid-19-in-washington-st...
Doing this for just one state was a pretty substantial effort. I imagine there are multiple people at the Times who are spending several hours a day reviewing and cleaning scraped data (seems every couple of days some formatting change breaks your scripts, or a source publishes data that later needs to be retracted).
The Times dataset appears to contain per-county case and death observations in a time series, going all the way back to the first confirmed U.S. case in January in Snohomish County, WA. This makes it by far the most comprehensive time series dataset of U.S. COVID-19 cases publicly available.
Some people in this thread linked to the Johns Hopkins CSSE dataset; I've looked at this data but it doesn't go back very far in time for the U.S., and the tables are published as daily summaries with differing table schemas which makes them hard to use out of the box. For some days earlier in March, "sublocations" aren't even structured (for example the same column contains, "Boston, MA" and "Los Angeles County", making it very hard to use). No disrespect to the team behind the JHU dataset; it attempts to cover the whole world since the outbreak began which is an incredible and difficult goal. But for mapping and studying the outbreak in the U.S., the Times dataset will likely be the best choice right now.
Huge kudos to the New York Times team for making this data freely available.
So before I duplicate any effort, has anyone found a tracker that takes this dataset's death counts and updates a Pueyo/Khan Academy-style graph [1], at different granular levels (world, nation, province/state)? Pueyo's analysis technique uses deaths each day to impute a back-looking inferred actual infected count instead of relying upon reported testing numbers, and that in turn can be used to infer a two-week forward-looking infected and death count based upon currently-known stats on average time to hospitalization and mortality. After two weeks, it gets increasingly more speculative, but by the time you have number of known deaths today, you almost have a baked-in outcome two weeks from now.
[1] https://www.youtube.com/watch?v=mCa0JXEwDEk
[1] https://www.youtube.com/watch?v=mCa0JXEwDEk
Not this data, but I like Johns Hopkins rendering of their data:
https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.h...
https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.h...
This site has some of what you want.
https://wuflu.live
https://wuflu.live
Doesn't that method assume we know the mortality rate, which we absolutely do not right now? It also would fail to account for any kind of containment measures.
[deleted]
god damn it NYT, "New York City" is not a county. Why can't I get borough by borough breakdowns? (each borough is its own county)
edit: someone else posted this, which DOES give county-by-county breakdowns: https://accelerator.weather.com/bi/?perspective=dashboard&pa...
edit: someone else posted this, which DOES give county-by-county breakdowns: https://accelerator.weather.com/bi/?perspective=dashboard&pa...
Is there anywhere (global) data about total number of tests made? The plain number of positive cases is quite difficult to make sense of.
For the US, the COVID Tracking Project have tested, positive, negative numbers for most states.
They also have the dataset available as a spreadsheet or a JSON or CSV API:
https://covidtracking.com/data/
The project was started by some journalists from The Atlantic. Seems to be a lot of duplicated effort in this area, although perhaps they are sharing data.
https://covidtracking.com/about-team/
They also have the dataset available as a spreadsheet or a JSON or CSV API:
https://covidtracking.com/data/
The project was started by some journalists from The Atlantic. Seems to be a lot of duplicated effort in this area, although perhaps they are sharing data.
https://covidtracking.com/about-team/
https://ourworldindata.org/coronavirus-testing-source-data
The key metric is tests/confirmed case.
When I've looked, this is very predictive of confirmed cases (2 weeks ago) per death.
The key metric is tests/confirmed case.
When I've looked, this is very predictive of confirmed cases (2 weeks ago) per death.
That was the best I had also found, but it is not of decent quality to me. sources seem to be quite out of date and often based on news articles, not actual daily data.
I've been downloading this daily and watching some things. Mostly deaths in the US:
https://www.ecdc.europa.eu/en/publications-data/download-tod...
https://www.ecdc.europa.eu/en/publications-data/download-tod...
According to [checks notes] the internet, you can't make this observation. If you do, you want people to die!!
> In light of the current public health emergency, The New York Times Company is providing this database under the following free-of-cost, perpetual, non-exclusive license. Anyone may copy, distribute, and display the database, or any part thereof, and make derivative works based on it, provided (a) any such use is for non-commercial purposes only and (b) credit is given to The New York Times in any public display of the database, in any publication derived in part or in full from the database, and in any other public use of the data contained in or derived from the database.
How is this form of license valid for what's essentially just numbers from publicly available data? Do I get to take numbers from various sources and then require credit and that nobody can use them for commercial purposes?
How is this form of license valid for what's essentially just numbers from publicly available data? Do I get to take numbers from various sources and then require credit and that nobody can use them for commercial purposes?
"Just numbers" well not quite. They are data which can be interpreted as information. This is like saying books are just a collection of letters.
This link may be informative and provide a US context. Sorry I don't have a more content specific reference.
https://academia.stackexchange.com/questions/63139/public-da...
This link may be informative and provide a US context. Sorry I don't have a more content specific reference.
https://academia.stackexchange.com/questions/63139/public-da...
Phone books are actually legally just a collection of numbers and are not considered a creative work, because there is nothing creative about the mere act of collating them. Thus, they cannot be copyrighted.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
If NYT provides a "value add" here beyond mere aggregation and deduplication they are probably copyrightable. Otherwise they probably don't hold a copyright here (although they may attempt to claim it, just like RTS did).
https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
If NYT provides a "value add" here beyond mere aggregation and deduplication they are probably copyrightable. Otherwise they probably don't hold a copyright here (although they may attempt to claim it, just like RTS did).
Your personal data is also just numbers. But it does seem weird. These numbers are already only authoritatively available from the government health services, right? The New York Times doesn't own the data either, if it's public it's public, they're providing a convenience. It's actually pretty analagous to news: they don't own the fact that an event happened, but they own their wording and they provide a service in getting that wording to you.
edit: Yeah, "Over months, our journalists have recorded the details on newly confirmed cases as reported by state and local officials." It's all government, just aggregated by NYT instead of the federal government.
edit: Yeah, "Over months, our journalists have recorded the details on newly confirmed cases as reported by state and local officials." It's all government, just aggregated by NYT instead of the federal government.
I think it's just the Times firing a shot across the bows of other newspapers who will re-publish the data as their own without attribution.
Newspapers sometimes do jerky things like that.
Newspapers sometimes do jerky things like that.
Oh that makes sense. The source of the data aside, the fact that it's not aggregated by the Feds means they did do a shit ton of work here...
Depends on the jurisdiction. In the US I'd be surprised if this compilation met the threshold of originality needed.
See Feist v. Rural for the Supreme Court decision on copyrights over compilations of data.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
See Feist v. Rural for the Supreme Court decision on copyrights over compilations of data.
https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
Have you tried tracking these numbers even at the state level? The data entry/scraping aspect alone is substantial, but when you get down to the nitty gritty of the wildly varying ways that each agency (state and county) reports these numbers, then you're essentially required to have an opinionated framework and domain knowledge to maintain a consistent, canonical dataset.
Sweat of the brow does not establish copyright in the United States.
Software is just numbers too.
Probably not. To the extent that the numbers simply reflect facts about the world and don't involve any creativity on the part of the New York Time they aren't subject to copyright.
EDIT: Some more details https://copyright.uslegal.com/enumerated-categories-of-copyr...
EDIT: Some more details https://copyright.uslegal.com/enumerated-categories-of-copyr...
How is this form of license valid for what's essentially just numbers?
Are you saying that the NYT has added no value over just an unformatted stream of raw numbers?
Are you saying that the NYT has added no value over just an unformatted stream of raw numbers?
In the US at least the standard for copyright is originality. Value add or "sweat of the brow" doesn't come into it.
All data is just numbers. A book is just numbers. Software is just numbers.
If you can have a license on anything, you have to be able to license numbers.
The law isn't like pure logic... not all numbers are the same, even if they are the same number.
What colour are your bits? https://ansuz.sooke.bc.ca/entry/23
If you can have a license on anything, you have to be able to license numbers.
The law isn't like pure logic... not all numbers are the same, even if they are the same number.
What colour are your bits? https://ansuz.sooke.bc.ca/entry/23
Yes.
My pet peeve with nearly all maps showing this data is that they are on geographical maps. A virus spread is only a weakly geographical phenomenon - most infections happen through human-to-human contact so it makes more sense to show on a population-weighted cartogram. Has anyone made a map like that? A great example is the 538 hexagonal-tile electoral college map: https://projects.fivethirtyeight.com/2016-election-forecast/
or one of these population-weighted projections of different areas:
http://news.bbc.co.uk/2/hi/in_pictures/8284655.stm
or one of these population-weighted projections of different areas:
http://news.bbc.co.uk/2/hi/in_pictures/8284655.stm
Kudos to NYT, but I think the COVID Tracking Project data is probably better because it attempts to measures total testing as well (positives and negatives). I've been using it to report state-level testing statistics and new case/death curves:
https://www.deptofnumbers.com/covid19/
From the data I've learned that Washington state appears to be getting their arms around this thing:
https://www.deptofnumbers.com/covid19/washington/
https://www.deptofnumbers.com/covid19/
From the data I've learned that Washington state appears to be getting their arms around this thing:
https://www.deptofnumbers.com/covid19/washington/
https://github.com/CSSEGISandData/COVID-19
The NYT spent more effort patting themselves on the back than they did compiling the data.
The NYT spent more effort patting themselves on the back than they did compiling the data.
Their data is of higher quality than upstream sources like the Johns Hopkins CSSE set. Notably, it's in long-form (one record per day per area) instead of wide-form and the schema is consistent.
This takes some know-how and a bunch of ETL effort. There is significant value in their transformation of upstream data.
This takes some know-how and a bunch of ETL effort. There is significant value in their transformation of upstream data.
That map looks good from a global level, but when you zoom in, it kind of falls down.
For example, two towns in my state have confirmed cases, are ported by the state health authorities and the local newspapers, but neither is on that map.
For example, two towns in my state have confirmed cases, are ported by the state health authorities and the local newspapers, but neither is on that map.
usafacts.org has daily county data from 1/23 forward.
For those interested in data, the COVID Tracking Project (http://covidtracking.com/) doesn't have county data, but it does have test data (both total, positive, and negative). Very helpful given how relatively useless all of these case counts are without knowing how many tests were conducted.
I quickly hacked a website together this weekend based on the COVID Tracking Project data because I wanted to see how much testing was being done and where.
I think that it's pretty interesting to sort by the columns and browse the data.
The site's not great on mobile, but I'm working on that this weekend...
http://VirusTracking.net
I think that it's pretty interesting to sort by the columns and browse the data.
The site's not great on mobile, but I'm working on that this weekend...
http://VirusTracking.net
GitHub link in case NYT is blocked at your... home-office?
https://github.com/nytimes/covid-19-data
https://github.com/nytimes/covid-19-data
[deleted]
Their data set shows only 1 case in san mateo county. That aint right
And places it behind a login page, requiring people to sign up to see it.
Here's what I don't understand... If viruses and bacteria are all over your body and do some useful things, how can some be bad? What separates 'bad' virus from a good virus?
Is that really a question? If some people are good and do useful things, how can there be murderers?
{ edit: explanation } Viruses are tiny fragments of dna and proteins that are only there to throw a monkey wrench into your cells. Sometimes that monkey wrench does fairly harmless things. These viruses thrive, because they find lots of healthy hosts over a long period of time.
But some break things badly, reproduce out of control or change your cells' functions to be less optimal. These don't do as well, since their hosts don't thrive.
The viruses don't 'care' what they're doing; they're just bits of junk that mutate a lot. The ones that kill things quickly, tend to die out.
The only reason the world isn't full of horrible murdering viruses (any more than it is) is, statistics. Not some grand plan or kindly act of nature.
{ edit: explanation } Viruses are tiny fragments of dna and proteins that are only there to throw a monkey wrench into your cells. Sometimes that monkey wrench does fairly harmless things. These viruses thrive, because they find lots of healthy hosts over a long period of time.
But some break things badly, reproduce out of control or change your cells' functions to be less optimal. These don't do as well, since their hosts don't thrive.
The viruses don't 'care' what they're doing; they're just bits of junk that mutate a lot. The ones that kill things quickly, tend to die out.
The only reason the world isn't full of horrible murdering viruses (any more than it is) is, statistics. Not some grand plan or kindly act of nature.
Viruses or bacteria aren't 'bad' or 'good' in themselves. The former is arguably 'life' and the latter is a single cell organism. Free will or morality are foreign concepts as to how they behave. 'good', 'bad' or 'successful' are labels we, humans, attribute to them.
Micro organisms are all about spreading genetic material, since they are so small, they have evolved passive strategies to do just that. Such as using a host organism and hitching a ride via fluid exchange.
What makes all the difference is where they will end up, and how they behave in particular environments. Coronavirus outside your body is harmless, but if it gets in your lungs, it may destroy those while it replicates.
Your gut teems with bacteria. There are dozens of species living in your colon. And you actually need them since they help break down food into its constituent parts in order for your gut to be able to absorb the nutrients.
https://en.wikipedia.org/wiki/Human_gastrointestinal_microbi...
However, if your bowels get perforated and that bacterial contents seeps between your organs, you'll get in deep deep trouble: these bacteria will start to infect organs they shouldn't infect, get into your bloodstream. You'll get sepsis and risk death.
So, harmless bacteria turn deadly if they end up in the wrong spot.
The interesting part about viruses is that through the way they operate - injecting itself into the DNA of our cells which will then bang off new copies of the virus - their genetic material has embedded itself into our own genome. It is estimated that 5 to 8% of the human genome consists of "endogenous viral elements". These are basically viral left overs, not viruses themselves, to be clear!
https://en.wikipedia.org/wiki/Endogenous_retrovirus
Micro organisms have been with us for a long, long time. We have evolved concurrently with them. And we will keep evolving alongside, evolving strategies - such immunity, or our ability to understand how they work - to ward off infections while new species will pop up and some might be successful in thwarting our evolved defenses.
Micro organisms are all about spreading genetic material, since they are so small, they have evolved passive strategies to do just that. Such as using a host organism and hitching a ride via fluid exchange.
What makes all the difference is where they will end up, and how they behave in particular environments. Coronavirus outside your body is harmless, but if it gets in your lungs, it may destroy those while it replicates.
Your gut teems with bacteria. There are dozens of species living in your colon. And you actually need them since they help break down food into its constituent parts in order for your gut to be able to absorb the nutrients.
https://en.wikipedia.org/wiki/Human_gastrointestinal_microbi...
However, if your bowels get perforated and that bacterial contents seeps between your organs, you'll get in deep deep trouble: these bacteria will start to infect organs they shouldn't infect, get into your bloodstream. You'll get sepsis and risk death.
So, harmless bacteria turn deadly if they end up in the wrong spot.
The interesting part about viruses is that through the way they operate - injecting itself into the DNA of our cells which will then bang off new copies of the virus - their genetic material has embedded itself into our own genome. It is estimated that 5 to 8% of the human genome consists of "endogenous viral elements". These are basically viral left overs, not viruses themselves, to be clear!
https://en.wikipedia.org/wiki/Endogenous_retrovirus
Micro organisms have been with us for a long, long time. We have evolved concurrently with them. And we will keep evolving alongside, evolving strategies - such immunity, or our ability to understand how they work - to ward off infections while new species will pop up and some might be successful in thwarting our evolved defenses.
Viruses can't reproduce on their own; they need to invade host cells and reprogram them to generate more virus. As potential hosts, we would consider a "bad" virus to be one that does this too aggressively, sickening or even killing the host.
The question seems pretty basic, but it's a good one. I realized I'm not 100% confident in the answer I was about to give. Hopefully we'll get an answer from an actual expert.
Some berries taste great but some berries poison you. Real puzzling isn't it.
It is interesting though. Why would a plant evolve to kill animals when the purpose of a berry is to spread seed through consumption?
Holly berries taste delicious to birds, but are slightly toxic to mammals. (enough to cause nausea, vomiting, and diarrhea, but not to kill you) Birds don't have receptors for capsaicin, the chemical that makes chilies spicy, which makes most mammals (but not humans) not eat them. Deadly nightshade isn't toxic to birds.
Birds tend to travel farther during any given time period than (typically terrestrial) mammals do, making them better at spreading seeds over a wide area. So it's evolutionarily advantageous for plants to optimize their fruits to be eaten by birds instead of mammals.
Birds tend to travel farther during any given time period than (typically terrestrial) mammals do, making them better at spreading seeds over a wide area. So it's evolutionarily advantageous for plants to optimize their fruits to be eaten by birds instead of mammals.
It is pretty interesting. I think if you remember that it's just random the way things get selected for, it's a little easier to understand. It's not like the plants decided over a generation or two to start killing things. In a lot of cases, birds could eat the berries and be fine and continue to spread the plant all over.
> purpose of a berry is to spread seed through consumption
That's not the purpose of the berry. The purpose is just to ensure the survival of the seed. In drought-stricken environments, the seed is probably better off keeping its shell of water and fructose until it can form a root system.
That's not the purpose of the berry. The purpose is just to ensure the survival of the seed. In drought-stricken environments, the seed is probably better off keeping its shell of water and fructose until it can form a root system.
One possible explanation is that many plants and berries which are poisonous to humans are not poisonous to other animals.
For other plants it may be advantageous to be poisonous to one species and not another
For other plants it may be advantageous to be poisonous to one species and not another
https://github.com/nytimes/covid-19-data