The recent breakthrough of llm’s solving major open problem’s in math is a direct contradiction to the article.
But, there is some truth to the article and perhaps it is more true in chat based interactions. The agentic, hands-off mode might tell a different story.
Nice find! The final paragraph of the Conclusion is amazingly prescient!
"Significantly, we found that translation quality as
indicated by BLEU score continues to improve with
increasing language model size, at even the largest
sizes considered. This finding underscores the value
of being able to train and apply very large language
models, and suggests that further performance gains
may be had by pursuing this direction further."
You should read that essay! The model is capable of producing phd level research on it's own with minimal set of prompt's. I was sad because of this paragraph:
"That view is that there is still a great deal of value in struggling with a mathematics problem, but that the era where you could enjoy the thrill of having your name forever associated with a particular theorem or definition may well be close to its end. So if your aim in doing mathematics is to achieve some kind of immortality, so to speak, then you should understand that that won’t necessarily be possible for much longer — not just for you, but for anybody."
He may seem to imply the end is only for some subset of reasons but if you read the entire essay he is just trying to give hope where the rest of the essay is really damning!
I am glad to see this today - after reading Tim gower's recent post on chatgpt 5.5 pro's phd level research ability I was feeling slightly sad about the future of math research.
Interestingly enough, the moment I saw the title I thought of Bill Thurston's famous article "On proof and progress in mathematics" and the top comment on the OP's thread is from him! Reading his reply sort of gave me the antidote to the temporary blues I felt yesterday.
.claude has become the new dotfiles. And what do people do when they want to start using dotfiles ? they copy other’s dotfiles and same is happening here :)
Try mimalloc. I have prototyped a feature on top of mimalloc and while effort was a dead end, the code (this was around 2020) was nicely written and well maintained and it was fun to hack on it. When I swapped jemalloc in our system with mimalloc, it was on par if not better when it comes to fragmentation growth control and heap usage perspective.
My original question is to understand why it is considered as huge tolerance and what should be considered low tolerance. I am suspecting the paper’s intention is not to compare apples and oranges. They are trying to optimize fp32 baseline by sometime resorting using fp16 as long as the resultant solution’s numerical accuracy is within thr tolerance level. They are going for the “low hanging fruits” type of optimization.
Why do you think it is a huge tolerance ? (Just curious since it is not clear to me if that will lead to too much of reduction in numerical accuracy compared to the speedup)
I haven't read the article yet but the moment I saw the writer is "Jordana Cepelewicz" I knew it is going to be great! I have read few of her past articles and they were excellent think pieces.
I have been programming in C professionally for the past ~4 years on a multi-threaded application in the ads industry and that is central to the business. C has lot of documented "foot-guns" but most of it (if not all of it) can be safeguarded against mis-use using "support structures" around the codebase like having standard code style (enforced via tools as much as possible), static code analysis (via compiler and external code scanning tools), dynamic code analysis (via Asan, Ubsan etc), having small, medium and large tests with right amount of code coverage etc. In addition to that, having a standard set of libraries (c modules for high performance data-structures/algorithms, macro based templates to work with types) and threading model goes a long way in reducing the pain to a very bare minimum. On the plus side, you enjoy a "simpler" language syntax that is easy to learn and with the above “support structures” one can become productive in no time. Plus there are newer books in the market that can teach you C properly, one I have read and recommend is "Effective C by Robert C. Seacord" - the author is one of the C standards committee members so you can't go wrong with the choice.
Ingress can also be costly especially if there is a steady state of high load traffic sent from on-prem machines to machines inside cloud (not sure about aws but have experienced this with azure where we had to resort to buy their expressroute which was very costly and ultimately unsustainable for us)
But, there is some truth to the article and perhaps it is more true in chat based interactions. The agentic, hands-off mode might tell a different story.