Factorization is key here. It separates dataset-level structure from observation-level computation so the model doesn't waste capacity rediscovering structure.
I've been arguing the same for code generation. LLMs flatten parse trees into token sequences, then burn compute reconstructing hierarchy as hidden states. Graph transformers could be a good solution for both: https://manidoraisamy.com/ai-mother-tongue.html
From the paper: "sleep insufficiency was significantly associated with lower life expectancy when controlling for traditional predictors of mortality, with only smoking displaying a stronger association."
Both the US and China rely on uranium-based fission, but they seem to be diverging on their next bet. China is exploring thorium based fission, while US is leaning toward fusion [1]
Tsinghua scientists use light to do AI math extremely fast (12.5 GHz, trillionths of a second), enabling real-time decisions like trading or robotics. [1]
Ever since "Attention Is All You Need", I've been reading research papers directly instead of waiting for tech news coverage. My information supply chain has evolved from news sites as explainer to following experts on Twitter to ChatGPT these days. I'm experimenting with one more step: what if the papers themselves were memes?
For example, mapping AlexNet's 50-year journey to the Pirates of the Caribbean sinking ship scene [1]. Or using Sheldon's milking stool argument to explain transformer architecture [2]. The absurdity seems to make the concepts more memorable. Each meme has a quiz to dig deeper into the paper.
What do you think? Is humor a legitimate tool for learning about research papers, or does it undermine the seriousness of the work?
Three years ago, when we started making profit as a bootstrapped startup, I was stunned by how little money I could reinvest in my own company compared to a funded competitor. We paid ourselves 20% of the profit and paid 40% in taxes and reinvested the rest i.e. 40% into our business. Meanwhile, a VC-backed competitor could show losses and invest 100% of the revenue plus the $10 million or $50 million they raised from investors. In this essay, I explain you how screwed up the incentives are for bootstrapped companies and a solution to fix this:
Think Stripe Atlas for bootstrapped companies, a service that incorporates you in countries with favorable tax treatment for reinvestment, where you only pay taxes on founder distributions, not on profits you reinvest.
OP here. In my previous post [1], I argued that code generation is the kingpin behind reasoning models. The bottleneck is LLMs generate code lossily due to tokenization fragmentation and treating code like natural language instead of structured graphs. In this post I propose:
1. Parsing user prompts into input graphs (using controlled English like ACE)
2. Parsing code into output graphs (AST trees)
3. Using graph transformers to map input graphs → output graphs
Core hypothesis: eliminating tokenization fragmentation ("ladlen" → ["lad", "len"]) and preserving tree structure could improve FrontierMath accuracy from 26% to 35-40%. No benchmarks yet. Just theory and a plan to test the improvement.
I've built compilers, not transformers, so would love technical feedback on:
- Is tokenization & linear structure really the bottleneck in code generation, or am I missing bigger issues?
- Is 35-40% improvement plausible, or overly optimistic?
- For those working on graph transformers: what approaches look promising?
Personally, I’m subscribing to multiple AI services in the hope of increasing my productivity. I don’t think I’ve ever subscribed to this many SaaS products before. My expectation is if these tools are taking a share of my spending, its coming from services I would otherwise hire contractors or employees for.
Ah, got it! "Advanced settings" is hiding below "Browse connectors". I thought its available only for some select companies and wanted to find it out here. Thanks!
I’ve updated my article (replaced GPT-5 with ChatGPT-5 in this section):
When you ask the latest model, ChatGPT-5 to multiply two large numbers, it doesn't calculate. It generates Python code, executes it in a sandbox, and returns the result. Unlike ChatGPT-3, which at least attempted arithmetic internally (and often failed), ChatGPT-5 delegates computation to external tools. [1]
And added this note:
[1] There are 2 ways to multiply numbers in GPT-5:
- Python mode, which uses python sandbox as mentioned above
- No tool mode, which uses internal reasoning
Python mode is approximately 2x more accurate than no tool mode in FrontierMath (26.3% vs 13.5% accuracy on expert level math). Python mode is also 4x to 10x more cost effective than no tool mode.
The GPT-5 API uses no-tool mode by default (tools must be explicitly enabled in API calls), while ChatGPT UI likely uses Python mode by default since Advanced Data Analysis is enabled by default for all Plus, Team, and Enterprise subscribers. This creates a significant cost optimization for OpenAI in the consumer product, while API users bear the full cost of inefficient reasoning unless they manually configure tool use.
---
Thanks again for flagging the inaccuracy, Simon! If you think any part of this update still misrepresents the model behavior, I’d love your input.
For example, if you ask someone to hang a painting in an art gallery 12 feet from the floor using a 13-foot ladder:
- a worker will use the safety rule of staying 5 feet away from the wall. This is what GPT-3 does. [1]
- an engineer will apply the Pythagorean theorem. This is what o3 does.
- Pythagoras, seeing it for the first time, will derive the theorem. GPT-5 is nowhere close to that.
This climbing up the ladder of abstraction existed even before LLMs. DeepMind's AlphaGo learned from human games. But AlphaGo Zero and AlphaZero trained entirely through self-play and began uncovering new strategies across Go, chess, and shogi. So whether it's code, a game, or pseudocode, they're all metadata operating at the same level of abstraction.
wow, tool use seems to reduce total tokens by ~4 to 10× and cost by orders of magnitude. I wonder what the accuracy difference would be. I'm going to try multiplying larger and larger numbers to see how accuracy compares between tool use and pure reasoning.
Exactly. Both the theater script and code are metadata that manipulates entities: characters in a play or variables in memory. There's definitely abstract-level understanding emerging: that's why models can be trained on python, but write code in java. That could be instructions like pseudo-code or the hidden document/theater script you mentioned. That capability jump from GPT3 to o1 is real.
But my point is: pure metadata manipulation has hit a ceiling or is moving at a crawling pace since o1. The breakthrough applications (like agentic AI) still depend on the underlying model's ability to generate accurate code. When that capability plateaus, all the clever orchestration on top of it plateaus too.
That's interesting, though I wonder if what's "baked into the weights" is closer to intuition than reasoning. Once reasoning traces are distilled into weights, the model stops thinking through problems and starts pattern-matching answers. That feels more like a stochastic parrot with intuition than an analytical reasoner.
OP here. Thanks for the thoughtful reply. Curious if you’ve measured o1’s accuracy and token cost with tool use enabled vs disabled? Wondering if python sandbox gives higher accuracy and lower cost, since internal reasoning chains are longer and pricier.
[Edit] Its probably premature to argue without the above data, but if we assume tool use gives ~100% accuracy and reasoning-only ~90%, then that 10% gap might represent the loss in the probabilistic model: either from functional ambiguity in the model itself or symbolic ambiguity from tokenization?
Hi HN,
OP here. I'd appreciate feedback from folks with deep model knowledge on a few technical claims in the essay. I want to make sure I'm getting the fundamentals right.
1. On o1's arithmetic handling: I claim that when o1 multiplies large numbers, it generates Python code rather than calculating internally. I don't have full transparency into o1's internals. Is this accurate?
2. On model stagnation: I argue that fundamental model capabilities (especially code generation) have plateaued, and that tool orchestration is masking this. Do folks with hands-on experience building/evaluating models agree?
3. On alternative architectures: I suggest graph transformers that preserve semantic meaning at the word level as one possible path forward. For those working on novel architectures - what approaches look promising? Are graph-based architectures, sparse attention, or hybrid systems actually being pursued seriously in research labs?
― Epictetus