The short answer is that tracing is way, way easier to implement in a predictable and reliably performant way. This especially matters for distributed computation and automatic differentiation, two areas where JAX shines.
AST parsing via reflection means your ML compiler needs to re-implement all of Python, which is not a small language. This is a lot of work and hard to do well with abstractions that are not designed for those use-cases. (I believe Julia's whole language auto-diff systems struggle for essential the same reason.)
Glad to see that you can make ensemble forecasts of tropical cyclones! This absolutely essential for useful weather forecasts of uncertain events, and I am a little dissapointed by the frequent comparisons (not just you) of ML models to ECMWF's deterministic HRES model. HRES is more of a single realization of plausible weather, rather than an best estimate of "average" weather, so this is a bit of apples vs oranges.
One nit on your framing: NeuralGCM (https://www.nature.com/articles/s41586-024-07744-y), built by my team at Google, is currently at the top of the WeatherBench leaderboard and actually builds in lots of physics :).
We would love to metrics from your model in WeatherBench for comparison. When/if you have that, please do reach out.
ERA5 covers 1940 to present. That's well before the satellite era (and the earlier data absolutely has more quality issues) but there's nothing from 170 years ago.
+1 one of most common mistakes for PhD students is picking a project based on research interests rather than the adviser. I believe it is relatively uncommon to speak with former students of an adviser, but this is something that everyone entering a PhD should do.
Finding a good mentor -- someone whose values you agree with and who sets you up for career success -- is far more important than working on any particular topic of interest. The world is full of interesting research topics, and very few PhDs work in the precise area of their PhD research for their entire career.
I work on weather prediction, both with traditional simulation methods and machine learning. I have not seen any examples of cellular automata used for useful predictions in this space.
I guess "quantum physicist" can mean either an academic credential or a job title. In my case, I have the former but no longer the later. I finished my PhD nine years ago and no longer work in the field.
As former quantum physicist, I find it little troubling to read "quantum theory has reached a dead end" in specific reference to the interpretation of quantum mechanics. Most quantum physicists could not care less about how quantum mechanics is interpreted when it makes highly accurate quantitative predictions, and there are still plenty of interesting open problems for quantum theory (e.g., related to the practical design of algorithms and hardware for quantum computers).
This article also misses what is likely the leading interpretation of quantum mechanics by actual quantum physicists, namely that the measurment problem is solved by decoherence (the quantitative theory of how classical states emerge from quantum states):
My experience was that robo-investors are great until you need something special. Then they can become rather painful.
Exmaple: I got divorced last year. Betterment took weeks of time and many phones calls until they were able to figure out a way to divide our assets evenly, without a large difference in cost basis. Their automatic algorithm for dividing accounts just didn't know how to handle it.
If UBS figures out how to offer a higher level of service on top of robo-advising, that could be a real win.
Most of the "bugs" caught here (including in TensorFlow and in my own project, Xarray) seems to actually be typos in the test suite. This is certainly a good catch (and yes, linters should check for this!), but seems a little oversold to me.
NERSC is part of DOE’s Office of Science. Nuclear weapon development is done by DOE’s National Nuclear Security Administration (NNSA), which sponsors labs like Lawrence Livermore and Los Alamos. It definitely isn’t NERSC, NNSA has its own dedicated supercomputers for weapons work.
> Neural networks need completely different optimisation methods, and there is no practically useful application of any of the Newton or Quasi-Newton methods for their optimisation.
I don't think this is quite fair. There are several variations of 2nd order methods, notably KFAC and Shampoo, that seem to quite effective for large-scale neural network training, e.g., see the intro of this paper for an overview: https://openreview.net/forum?id=-t9LPHRYKmi
A PhD is certainly not for everyone (or even most), but it's rather unfair to say that it does not teach any employable skills. Even PhDs who go on to work in completely unrelated fields learn how to make progress on poorly defined problems and to advance the frontier of human knowledge. And a PhD is also of course a hard prerequisite for career in academia or research.
As for that survey of PhD students, I suspect you would get similarly dismal reviews of parenthood from parents of 0-5 year old children -- but of course that doesn't mean that nobody should have children! A better survey would ask PhD students how they feel about the experience well after they are done with it.
ECMWF makes probabilistic forecasts, in the form of an ensemble of 50 IID examples. So this is mostly matter of Windy figuring out how to put that information into their UI.
The general case of implicit differentiation, i.e., for functions y(x) defined by the constraints F(x, y(x)) = 0, where x and y are vectors, is solved by "implicit function theorem": https://en.wikipedia.org/wiki/Implicit_function_theorem
∂ y(x) = -(∂_1 F(x, y(x)))^{-1} (∂_0 F(x, y(x)))
where ∂ denotes partial differentiation.
This turns out to be an incredibly useful identity for calculating derivatives. No matter how you calculated a solution to the equation, computing derivatives is "just" a matter of performing a linear solve.
If the calculation you performed is a solution to solving an equation, implicit differentiation is typically much faster, less memory intensive and more accurate than calculating derivatives by differentiating through your solver. For examples, you might check-out a recent paper I co-authored with colleagues at Google: https://arxiv.org/abs/2105.15183
Does anyone know where the money is coming from? I'm guessing foundations like Schmidt Futures? It seems like financing is the crucial challenge for getting anything like this off the ground.
This post is yet another example of why you should never use APIs for random number generation that rely upon and mutate hidden global state, like the functions in numpy.random. Instead, use APIs that explicitly deal with RNG state, e.g., by calling methods on an explicitly created numpy.random.Generator object. JAX takes this one step further: there are no mutable RNG objects at all, and the users has to explicitly manipulate RNG state with pure functions.
It’s a little annoying to have to set and pass RNG state explicitly, but on the plus side you never hit these sorts of issues. Your code will also be completely reproducible, without any chance of spooky “action at a distance.” Once you’ve been burned by this a few times, you’ll never go back.
You might think that explicitly seeding the global RNG would solve reproducibility issues, but it really doesn’t. If you call into any code you didn’t write, it might also be using the same global RNG.
By the way, is there another reference with a full description of how the benchmarks are setup? I'd be curious how Xarray (end user API) + Dask (distributed compute) + Zarr (distributed storage) compares.
Xarray definitely takes a different philosophical approach based on its roots in the Python data science ecosystem, compared to "all in one" solutions like a full array databases.
AST parsing via reflection means your ML compiler needs to re-implement all of Python, which is not a small language. This is a lot of work and hard to do well with abstractions that are not designed for those use-cases. (I believe Julia's whole language auto-diff systems struggle for essential the same reason.)