Economics Nobel laureate Paul Romer is a Python programming convert(qz.com)
qz.com
Economics Nobel laureate Paul Romer is a Python programming convert
https://qz.com/1417145/economics-nobel-laureate-paul-romer-is-a-python-programming-convert/
11 comments
“If you asked me to predict what a quantitative economist would be using, it would be Python, followed by R, and maybe followed by Java or C, or something like that.”
I too was looking for good data. It’s hard to measure this as practical application doesn’t line up with publication.
I was surprised when I started working in health 10 years ago that the predominant tool was Excel, and then SAS. Even 5 years ago in health grad school, they only taught SAS.
This is slowly changing to R and python, but general data analysis skills is less than basic engineering stuff 20 years ago.
I too was looking for good data. It’s hard to measure this as practical application doesn’t line up with publication.
I was surprised when I started working in health 10 years ago that the predominant tool was Excel, and then SAS. Even 5 years ago in health grad school, they only taught SAS.
This is slowly changing to R and python, but general data analysis skills is less than basic engineering stuff 20 years ago.
Some departments in the FDA require clinical trial data to be submitted in SAS formats :-/
This is a myth:
http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html
https://www.fda.gov/downloads/forindustry/datastandards/stud...
https://www.fda.gov/downloads/forindustry/datastandards/stud...
No, by the text of that same link you provided: "... origin of this fallacy is probably related to the fact that data must be submitted in the XPT "transport format" (which was originally created by SAS)." While that post goes on to say, "This [XPT] data format is now an open standard." That is somewhat disingenuous. The XPT format requires IBM Mainframe floats and other wierdness. It's not always that easy to write XPT.
> I'm not an economist so maybe this is something about economics per se.
No, this is just a case of historical contingency, or the founder effect, depending on your preferred metaphor. Stata and Mathematica are suitable tools (and were suitable early on) that happened to be adopted by a few economists, whose choice was then spread and perpetuated via various organizations and institutions, word of mouth and curricula.
No, this is just a case of historical contingency, or the founder effect, depending on your preferred metaphor. Stata and Mathematica are suitable tools (and were suitable early on) that happened to be adopted by a few economists, whose choice was then spread and perpetuated via various organizations and institutions, word of mouth and curricula.
Nate Silver shared that a vast majority of the latest five thirty eight model for 2018 is constructed in Stata. Just an anecdote, but fairly practical example of something recently written that’s using that technology.
Depends on the field. In the social sciences, Mathematica and Stata are probably the most widely used tools. Might be that way for economics too, but I'm not sure.
I believe they prefer window like interfaces instead of scripting or programming your models.
Stata used to be only command line driven and probably still has that functionality ( I used it in 1997-2007) ; then again as a mathematician I prefer R. Or any computer algebra system.
Most quantitative economists do not use Python. They use Stata, SAS, EViews, etc. For methods not yet implemented, the go to application is Matlab (matrix). Python has been gaining traction in the past five years however.
Academy is really slow to pick up the trend. And when one group collect years of legacy codes it is even harder to change
My grad program taught SAS, to my dismay.
I'm not an economist so maybe this is something about economics.
Stata seems to be really popular in economics. Certainly when I was getting my degree all the statistics, modelling and econometric courses at the economics department used either Excel or Stata. A few 'weird' kids used matlab, but the words "Python" or "R" where never mentioned.
Stata seems to be really popular in economics. Certainly when I was getting my degree all the statistics, modelling and econometric courses at the economics department used either Excel or Stata. A few 'weird' kids used matlab, but the words "Python" or "R" where never mentioned.
> If you asked me to predict what a quantitative economist would be using, it would be Python, followed by R, and maybe followed by Java or C, or something like that.
I've never heard of an economist using Java, and very few using C. Fortran is still quite popular in some areas.
Stata is extremely popular in some areas. GAUSS and Matlab in others (though GAUSS is declining for sure). R is quite popular, particularly since RStudio came along.
I've never heard of an economist using Java, and very few using C. Fortran is still quite popular in some areas.
Stata is extremely popular in some areas. GAUSS and Matlab in others (though GAUSS is declining for sure). R is quite popular, particularly since RStudio came along.
I've spent most of my career working with economists, and they've got a strong preference for Stata. It's what most of them learned in university. I'm hopeful that it's a preference that will slowly be replaced with Python--we're already making that change at the think tank where I work.
When I was finishing up grad school in 2013, I talked to a few econ grad students about what tools they were using for statistics and computing. It sounded like the people just starting out were generally familiar with and in favor of R, but among the older students it wasn't quite as popular. I don't remember anyone mentioning Python at the time.
Does anyone have any insight into why Paul Romer is a Julia skeptic?
In the blog post
https://paulromer.net/jupyter-mathematica-and-the-future-of-...
he says:
"Which reminds me. If you are a Julia enthusiast, how do you suppose the investors in this new language plan to make their big score?"
Also there's this:
https://twitter.com/paulmromer/status/985507319114096640
It doesn't seem like he's followed up on his threat/promise to write a blog post about why he's not a Julia enthusiast, however.
In the blog post
https://paulromer.net/jupyter-mathematica-and-the-future-of-...
he says:
"Which reminds me. If you are a Julia enthusiast, how do you suppose the investors in this new language plan to make their big score?"
Also there's this:
https://twitter.com/paulmromer/status/985507319114096640
It doesn't seem like he's followed up on his threat/promise to write a blog post about why he's not a Julia enthusiast, however.
I use Python and it's OK, but I find a lot of the breathless hype around it as a language a little baffling as there are more than an average number of stupidities in there. However this piece is really around exploiting the amazing infrastructure that has built up around Python that empowers mathematical and statistical research, which is fair enough.
> As there are more than an average number of stupidities there.
Please post more meaningful comments. Sure it doesn't have static typing or performance of C, but it has a stable framework in every conceivable application (Web Dev, machine learning, CLI, you name it) and works as a top notch scripting language. This is not "stupidity", it was by design to make it batteries included and easy to use with a tradeoff with the above things.
Please post more meaningful comments. Sure it doesn't have static typing or performance of C, but it has a stable framework in every conceivable application (Web Dev, machine learning, CLI, you name it) and works as a top notch scripting language. This is not "stupidity", it was by design to make it batteries included and easy to use with a tradeoff with the above things.
What are these "stupidities" that bother you?
Personally, I use Python a lot for anything related to data science. No language is perfect, but I'd say Python has a lot less deficiencies and warts than most other mainstream languages, and it's extremely well suited for tasks in data science and related fields such as machine learning.
The main issue with Python is that its default platform (CPython) isn't very efficient. That's not a problem in the language itself; in fact, it's partly caused by all the benefits of a high-level language: you simply don't have the same facilities to optimize your resource consumption as you do in, say, Rust.
The upside is of course that the code is far more concise and readable.
Personally, I use Python a lot for anything related to data science. No language is perfect, but I'd say Python has a lot less deficiencies and warts than most other mainstream languages, and it's extremely well suited for tasks in data science and related fields such as machine learning.
The main issue with Python is that its default platform (CPython) isn't very efficient. That's not a problem in the language itself; in fact, it's partly caused by all the benefits of a high-level language: you simply don't have the same facilities to optimize your resource consumption as you do in, say, Rust.
The upside is of course that the code is far more concise and readable.
Whilst that is true, for data science related things I end up using Pandas (and Numpy) often which are fast.
So most of the heavy lifting is not done by python itself.
So most of the heavy lifting is not done by python itself.
Yes, that's the case for most data scientists, and often for users in general. You can get pretty good performance levels from Python by delegating to C correctly. Even end users tend to do that because the data science stack of Python (numpy and the many libraries using it) spend almost all of their runtime in C routines.
If you are using Numpy correctly (even with little understanding of Python itself and its C-API) then virtually all your hot loops are executed in optimal C code.
If you are using Numpy correctly (even with little understanding of Python itself and its C-API) then virtually all your hot loops are executed in optimal C code.
So what? Pandas and numpy are still use with Python code using their APIs. The implementation details are not something the users care about. They write Python.
Not sure what you're arguing. If you use numpy and pandas properly, most of your computations happen in optimized C routines.
So you will write Python, but get very good resource (both time and space) efficiency.
So you will write Python, but get very good resource (both time and space) efficiency.
Here's a classic:
def a(arg = []):
return arg
a([1]) #==> [1]
arr = a() #==> []
arr.append(1)
a() #==> [1]That's slightly surprising, but I wouldn't call it a wart. A proper wart is something that bothers you even after years of using the language, not something that surprises you a couple of times as a beginner, but then ceases to be a problem - or indeed, in this case, has some good uses.
The obvious alternative in this case would be to reconstruct the default value. This doesn't work very well with the language's evaluation model - since the function definition is only evaluated once, and it's very inefficient and in fact surprising to perform later re-evaluations.
Furthermore, this behavior can be useful. If you don't want it, you just use `None` as the default and then assign in the function body. However, if you do want it, there would be no other way to get it, and no similarly easy workaround.
Finally, had there been a consensus of it being a wart, it would have been removed in Python 3.x. It wasn't.
The obvious alternative in this case would be to reconstruct the default value. This doesn't work very well with the language's evaluation model - since the function definition is only evaluated once, and it's very inefficient and in fact surprising to perform later re-evaluations.
Furthermore, this behavior can be useful. If you don't want it, you just use `None` as the default and then assign in the function body. However, if you do want it, there would be no other way to get it, and no similarly easy workaround.
Finally, had there been a consensus of it being a wart, it would have been removed in Python 3.x. It wasn't.
>I use Python and it's OK, but I find a lot of the breathless hype around it as a language a little baffling as there are more than an average number of stupidities in there.
There are many warts that will never go away simply because they're embedded so much within the culture of the language and because undoing the wart would break so much working code.
I love python and I've used it every day for 15 years but there are core parts of the language I hate with a passion.
If in 3.9:
* "if x" started failing with an exception for most non-boolean values of x (e.g. lists, strings, numbers, etc.)
* strings stopped being iterable by default
then I'd be celebrating, but I know it's never going to happen.
There are many warts that will never go away simply because they're embedded so much within the culture of the language and because undoing the wart would break so much working code.
I love python and I've used it every day for 15 years but there are core parts of the language I hate with a passion.
If in 3.9:
* "if x" started failing with an exception for most non-boolean values of x (e.g. lists, strings, numbers, etc.)
* strings stopped being iterable by default
then I'd be celebrating, but I know it's never going to happen.
> "if x" started failing with an exception for non-boolean values of x
I would call that a personal preference, and one that is far from consensus, rather than a "wart".
A wart is a known problem in the language, that most users consider to exert a negative impact on their usecases. Example would be Type Erasure in Java, which is there for historical reasons, can not be fixed, and is generally agreed by Java programmers to be a Bad Thing.
There isn't anything like this consensus for the changes you are suggesting.
All other mainstream languages support use of non-booleans as predicates, including for example C++ and Java, which are statically typed.
This change would make Python more strict about typing than virtually all mainstream static languages! Given how Python is dynamically typed, I'd suspect this change would be extremely unpopular.
> strings stopped being iterable by default
Again, not something most people would use. There are many fields in which strings are used to store sequences of characters where each character has a meaning. In all those fields, `for char in string` is a very useful idiom.
I'm not sure how this is a wart or what's your usecase that string iterability is breaking. Indeed, strings being iterable is in line with the general iterability philosophy and unlikely to change (which is a good thing for most users).
I would call that a personal preference, and one that is far from consensus, rather than a "wart".
A wart is a known problem in the language, that most users consider to exert a negative impact on their usecases. Example would be Type Erasure in Java, which is there for historical reasons, can not be fixed, and is generally agreed by Java programmers to be a Bad Thing.
There isn't anything like this consensus for the changes you are suggesting.
All other mainstream languages support use of non-booleans as predicates, including for example C++ and Java, which are statically typed.
This change would make Python more strict about typing than virtually all mainstream static languages! Given how Python is dynamically typed, I'd suspect this change would be extremely unpopular.
> strings stopped being iterable by default
Again, not something most people would use. There are many fields in which strings are used to store sequences of characters where each character has a meaning. In all those fields, `for char in string` is a very useful idiom.
I'm not sure how this is a wart or what's your usecase that string iterability is breaking. Indeed, strings being iterable is in line with the general iterability philosophy and unlikely to change (which is a good thing for most users).
>I would call that a personal preference
shrug it's two features that I see causing two classes of bugs on an extremely regular basis.
It's a trade off that affects everybody, but it's hard to weight up the trade off without a lot of experience.
>There isn't anything like this consensus for the changes you are suggesting.
Exactly what I said, yes...
>This change would make Python more strict about typing than virtually all mainstream static languages!
I think dialing up the strictness is a good thing and so do the language designers (that was the idea behind introducing type hints).
>Again, not something most people would use.
That's exactly the point. Making strings iterable list of characters isn't something that most people use - its primary function is to cause weird bugs for functions that take lists of strings (and you get an output like 'm', 'y', 'd', etc.).
Changing it so that you'd have to do string.chars to get an iterable list would, at very little cost, cut out a pretty common class of bug.
shrug it's two features that I see causing two classes of bugs on an extremely regular basis.
It's a trade off that affects everybody, but it's hard to weight up the trade off without a lot of experience.
>There isn't anything like this consensus for the changes you are suggesting.
Exactly what I said, yes...
>This change would make Python more strict about typing than virtually all mainstream static languages!
I think dialing up the strictness is a good thing and so do the language designers (that was the idea behind introducing type hints).
>Again, not something most people would use.
That's exactly the point. Making strings iterable list of characters isn't something that most people use - its primary function is to cause weird bugs for functions that take lists of strings (and you get an output like 'm', 'y', 'd', etc.).
Changing it so that you'd have to do string.chars to get an iterable list would, at very little cost, cut out a pretty common class of bug.
Function definitions are evaluated once and early. Easy rule to learn and not really problematic.
I'm surprised people needed to ask, and I was very specifically talking about the language itself rather than the libraries (so "batteries included" is not a counterargument); but here is a small non-exhaustive assortment of nonsense in Python:
1. Does 3/2 == 1 or 1.5 ? Either answer would be logical and OK, provided you stick to it. The utterly daft answer that Python gives is "it depends on what version of the language you are using, and/or whether you do "from __future__ import division""
2. Generalising 1, a painful set of breaking changes between 2 and 3 that could have largely been avoided by aliasing names or introducing new names for changed concepts / methods; eg make xrange = range; add a new function rather that redefine what 'print' is (and yes, failing to make 'print' a function in the first place was a shockingly bad decision). Read about the insane amout of effort that Dropbox put into migrating their codebase to see what a ballsup this all is - polite languages respect backwards compatibility.
3. The object system is a mess: type(some_obj_i_made) is <type 'instance'> - wat? - unless you explicitly inherit your class from object when it suddenly does something sensible; constructor inheritance is a confusing bog. Yes yes old style vs new style objects, but that's the problem. This stuff shouldn't be hard; see eg Ruby.
4. edit: scoping - oy vey
Look, it's OK. It works. The ecosystem makes up for an awful lot of this rubbish. But don't be so devoted to a technology that you're blind to its faults.
1. Does 3/2 == 1 or 1.5 ? Either answer would be logical and OK, provided you stick to it. The utterly daft answer that Python gives is "it depends on what version of the language you are using, and/or whether you do "from __future__ import division""
2. Generalising 1, a painful set of breaking changes between 2 and 3 that could have largely been avoided by aliasing names or introducing new names for changed concepts / methods; eg make xrange = range; add a new function rather that redefine what 'print' is (and yes, failing to make 'print' a function in the first place was a shockingly bad decision). Read about the insane amout of effort that Dropbox put into migrating their codebase to see what a ballsup this all is - polite languages respect backwards compatibility.
3. The object system is a mess: type(some_obj_i_made) is <type 'instance'> - wat? - unless you explicitly inherit your class from object when it suddenly does something sensible; constructor inheritance is a confusing bog. Yes yes old style vs new style objects, but that's the problem. This stuff shouldn't be hard; see eg Ruby.
4. edit: scoping - oy vey
Look, it's OK. It works. The ecosystem makes up for an awful lot of this rubbish. But don't be so devoted to a technology that you're blind to its faults.
>I use Python and it's OK, but I find a lot of the breathless hype around it as a language a little baffling as there are more than an average number of stupidities in there.
There's fewer stupidities than languages like javascript or perl and it ends up being more practical than languages like haskell or ocaml.
One of it's unappreciated facets is that it's good at language interop. Instead of trying to be all things to all people (e.g. like go) it just makes it easy to interop with C where you need speed and focuses on being a good high level language.
I think the surprise surrounding its popularity is indicative of the fact that few people really understand what makes a great language.
There's fewer stupidities than languages like javascript or perl and it ends up being more practical than languages like haskell or ocaml.
One of it's unappreciated facets is that it's good at language interop. Instead of trying to be all things to all people (e.g. like go) it just makes it easy to interop with C where you need speed and focuses on being a good high level language.
I think the surprise surrounding its popularity is indicative of the fact that few people really understand what makes a great language.
I'm surprised there seems to be have been no HN discussion on this years Economics prize. Or are my HS search skills lacking?
I haven't found any, either.
This is a shame because economics is already a computational discipline and increasingly so. It can be of interest to the HN crowd.
There is an awful lot of programming done by people that are not professional programmers. As an economist myself, I think code have become so central to our profession that we can no longer afford to ignore software engineering best practices. Conversely, there may be quite a few topics in computational economics that interest programmers.
For instance, an awful lot of macroeconomics - including to an extent Romer's sub-field, endogenous growth theory - is essentially programming in Dynare [0] and doing computer simulations.
Nordhaus got the prize for developing (quite basic, actually) simulation models that integrate the economic systems and environmental systems. See for instance there [1].
Micro-simulation is of course a big area where programming and economics meet - you can follow development on github [2, 3]. This is code that has an enormous impact on society, as it is used as a basis to evaluate the effects of policies.
Econometrics, aka statistics for economics, is also a big area where programmers could contribute to economic research. There is a big push to adapt ML techniques to solve economists' problems, eg Athey's research [4].
[0] http://www.dynare.org/
[1] https://sites.google.com/site/williamdnordhaus/dice-rice
[2] https://github.com/openfisca
[3] https://github.com/InseeFr/Destinie-2
[4] https://arxiv.org/abs/1510.04342
This is a shame because economics is already a computational discipline and increasingly so. It can be of interest to the HN crowd.
There is an awful lot of programming done by people that are not professional programmers. As an economist myself, I think code have become so central to our profession that we can no longer afford to ignore software engineering best practices. Conversely, there may be quite a few topics in computational economics that interest programmers.
For instance, an awful lot of macroeconomics - including to an extent Romer's sub-field, endogenous growth theory - is essentially programming in Dynare [0] and doing computer simulations.
Nordhaus got the prize for developing (quite basic, actually) simulation models that integrate the economic systems and environmental systems. See for instance there [1].
Micro-simulation is of course a big area where programming and economics meet - you can follow development on github [2, 3]. This is code that has an enormous impact on society, as it is used as a basis to evaluate the effects of policies.
Econometrics, aka statistics for economics, is also a big area where programmers could contribute to economic research. There is a big push to adapt ML techniques to solve economists' problems, eg Athey's research [4].
[0] http://www.dynare.org/
[1] https://sites.google.com/site/williamdnordhaus/dice-rice
[2] https://github.com/openfisca
[3] https://github.com/InseeFr/Destinie-2
[4] https://arxiv.org/abs/1510.04342
This is a great comment.
I believe the NY Fed has open source economic models in Julia, too.
I believe the NY Fed has open source economic models in Julia, too.
Maybe part of the problem is that unless you delve beyond layman level (and maybe even if you do?) economics is close-coupled to political views, and those start wars- so we tend to steer clear on this board.
The dismal science has earned the name many times over: no one wants to read a long-winded epistle on the relationship between bond yields and non-farm employment in Alabama between 1993 and 1998, and that's where a lot of the interesting parts of econ lie.
Likely because there aren't too many Economics majors on this forum (I am, BTW). From what I've seen on previous HN posts on economics topics, the discussion veers towards "Economics isn't a real science" variety, with the majority of the posts going back on forth on that comment, rather than the actual topic of the article.
People interested by the use of Python in economic research should check out QuantEcon [0], a project to develop tools and courses for computational economics [1] in Python and Julia
[0] https://quantecon.org/
[1] A fancy expression to say "using programming to solve economic models"
[0] https://quantecon.org/
[1] A fancy expression to say "using programming to solve economic models"
Yeah, and Economics Nobel laureate Thomas Sargent is a Julia programming convert (he even gave the JuliaCon 2016 keynote, video embedded below:)
https://juliacomputing.com/case-studies/thomas-sargent.html
EDIT to add: I should say that Thomas Sargent is one of the founders of QuantEcon (cited by OP), that's why I'm bringing this up here.
Another resource on Julia in Computational Economics:
https://juliaeconomics.com
https://juliacomputing.com/case-studies/thomas-sargent.html
EDIT to add: I should say that Thomas Sargent is one of the founders of QuantEcon (cited by OP), that's why I'm bringing this up here.
Another resource on Julia in Computational Economics:
https://juliaeconomics.com
(Cofounded by another Nobel laureate.)
> A fancy expression to say "using programming to solve economic models"
That fancy expression existed well before the birth of programming.
That fancy expression existed well before the birth of programming.
> he tried to use Mathematica to share one of his studies in a way that anyone could explore every detail of his data and methods. It didn’t work. He says that Mathematica’s owner, Wolfram Research, made it too difficult to share his work in a way that didn’t require other people to use the proprietary software, too
Sometimes I wonder where Mathematica would be if it were open sourced lets say in 2010. It had such a head start over Python.
Interestingly, Mathematica interop with Python is basically non-existent today, even worse than it was in the past. Short term moat building destroying long term viability.
Sometimes I wonder where Mathematica would be if it were open sourced lets say in 2010. It had such a head start over Python.
Interestingly, Mathematica interop with Python is basically non-existent today, even worse than it was in the past. Short term moat building destroying long term viability.
Not true: Python is callable from a Wolfram Notebook: https://reference.wolfram.com/language/tutorial/ConfigurePyt...
Romer also published notebooks and PDFs without issue previously: https://assets.aeaweb.org/assets/production/articles-attachm...
Something about his story is odd. Given his shady past at the World Bank, who knows what's actually true.
Romer also published notebooks and PDFs without issue previously: https://assets.aeaweb.org/assets/production/articles-attachm...
Something about his story is odd. Given his shady past at the World Bank, who knows what's actually true.
I loved using Mathematica in college. It was the first programming language I ever really used to make some cool things. I wish it was more accessible. I think one of the requirements of my dream job would be a work place that uses Mathematica.
It's really fun reading through "code golf" challenges where other languages take 10-15 lines for something that Mathematica has a built in for.
It's really fun reading through "code golf" challenges where other languages take 10-15 lines for something that Mathematica has a built in for.
Did you just say "code golf" and Mathematica in the same sentence?
Then you will love these. I own like 10 copies, which I give to people for their birthdays: http://adereth.github.io/blog/2017/11/02/playing-with-wolfra...
Then you will love these. I own like 10 copies, which I give to people for their birthdays: http://adereth.github.io/blog/2017/11/02/playing-with-wolfra...
Oh, those are quite cool. I'll have to get a pack sometime. I also think the Executed Today deck looks pretty cool (totally different card subject)
I used Mathematica a lot as an undergrad, but more or less exclusively for doing symbolic math, mostly tricky or tedious integrals.
Much later I realized that it's actually a more or less full-fledged programming language as well, but, I never got into that.
Much later I realized that it's actually a more or less full-fledged programming language as well, but, I never got into that.
[deleted]
We could recover Mathematica interop easily. i wrote a tool to convert Python expressions to Mathematica expressions, evaluate them, and return the results as Python expressions, probably back in 97. I believe M still has the Kernel C API so it should be easy to resurrect & make modern. http://library.wolfram.com/infocenter/MathSource/585/
Beyond the cool python angle, I find the following statement quite interesting: "...James Somers argued that Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." I've only been exposed to jupyter notebook references here and there...but i guess i should become a little more familiar with them.
While I think this is a good thing, an additional requirement is needed. The published notebook should be sequentially executed by an automated tool. Users can execute commands out of order and have artifacts from deleted commands. Without verification, you can be publishing notebooks with bugs in them that you don't see until you re-execute them.
Jupyter notebooks are ultimately just a text file, which can be version controlled, verified (via the VCS or independent hashing), and if necessary - digitally signed in a cryptographically secure manner.
Given that Git is already secure, I'd say all the researcher needs to know is basic usage of Git. Version controlling the notebook satisfies all the requirements you mentioned: prevents accidental distortion, is verifiable, restorable, etc.
Given that Git is already secure, I'd say all the researcher needs to know is basic usage of Git. Version controlling the notebook satisfies all the requirements you mentioned: prevents accidental distortion, is verifiable, restorable, etc.
It's not just tracability and security that's the issue, though. It's an issue of reproducibility. When I give you a notebook, how do you 'run' it to verify my calculations? You can't just press "run", because I might have executed the cells out of order, or deleted cells which later cells depended on. Version control doesn't solve this problem. Good practice might solve it, but it turns out that researchers are stunningly bad at following a strict top-to-bottom execution of notebooks. And who can blame them? The non-linearity of notebooks is one of their features. But when it comes to reproducibility and just the basic mechanics of sharing code, we definitely need a better solution.
You forget that a typical Python Jupyter notebook will import a dozen Python packages.
Versioning that it's still a major problem in the ecosystem.
Versioning that it's still a major problem in the ecosystem.
> Versioning that it's still a major problem in the ecosystem.
There's actually a perfectly good way to solve this problem: a virtual environment with a requirements.txt file.
There's actually a perfectly good way to solve this problem: a virtual environment with a requirements.txt file.
For research in particular, I'd suggest a Pipfile.lock. That will ensure that the packages installed match exactly, not just by version number.
[deleted]
Tools for combining the interactive work in notebooks with a more predictable linear flow are starting to mature. See for instance jupytext [1]: a bi-directional converter between jupyter notebooks and python/R/julia/...
I believe jupytext was featured here on HN a few weeks back.
[1] https://github.com/mwouts/jupytext
[1] https://github.com/mwouts/jupytext
I was recently asked by a company to do a take-home assignment, some data analysis stuff. I opted for a jupyter notebook as it sequentially allows you to both explore the data in an interactive manner while still programming in Python. It was great. I love notebooks.
"Jupyter notebooks may replace the traditional research paper typically shared as a PDF..." - hardly a replacement as long as Jupyter notebooks are not self-sufficient files you can download and view offline everywhere. Even if you have up-to-date versions of Python and all the libraries installed you still have to run a server and open it in a browser which is a way more clumsy than just opening a PDF in Okular/Evince/AcrobatReader. I also doubt Jupyter notebooks are easy to view on mobile devices.
You can view them just as easily as you would a pdf. Running the code inside them is what takes extra effort. Which isn’t necessary if you just want to understand the processes being applied.
Can they include pre-rendered code execution results to display when code execution is not available?
Yes, they do by default if I’m not mistaken. The output “cells” are saved alongside the code.
There's probably a lot of gnashing of teeth over this article at WRI/Mathematica.
As long as it makes Stephen Wolfram angry, I'm happy.
If you're a former employee, you know he is.
Next one will be a Julia convert.
According to this comment further down there already is one:
https://news.ycombinator.com/item?id=18174769
https://news.ycombinator.com/item?id=18174769
Someone should point him toward SageMath.
The article’s title is misleading. Doesn’t sound like he is a Python convert. Sounds like he is an open source tool convert.
The article’s title is misleading. Doesn’t sound like he is a Python convert. Sounds like he is an open source tool convert.
For example, this seems to set up most of the article:
"Economics involves a lot of math and statistics. The most commonly used tools to crunch numbers are the spreadsheet software Microsoft Excel and programming languages Stata and Mathematica."
Is this really true? Mathematica and Stata seem like established but niche products to me at this point. I wouldn't say either of them are "the most commonly used tools to crunch numbers."
If you asked me to predict what a quantitative economist would be using, it would be Python, followed by R, and maybe followed by Java or C, or something like that.
This was an interesting article in the sense I like learning these sorts of things about people, but the premise seemed off to me.
But I'm not an economist so maybe this is something about economics per se.