I always find Superintelligence inevitability interesting. And the author dismisses George Hotz argument about supply chains and the way that in order for ASI to truly take off it has to dominate not just the world of bits but atoms as well.
But I imagine there’s this problem that measuring intelligence in AI is not the same as measuring intelligence in humans. How is fable not already super intelligence. How is it not way smarter than the 99th percentile human? In terms of speed, broad ability and pattern matching behavior. If intelligence is loosely correlated with lifetime income, Anthropic billions in revenue should be a decent measure that LLMs are way smarter than most folks.
Because sometimes I think the inconvenient truth is that when people ask, you don’t think we’ll have AI that’s smarter than humans. It feels to me like we had AI that was smarter than humans a long time ago that somewhere around GPT three or GPT four we had AGI and if we already have AGI and we already have AI that’s smarter than humans then maybe the real question is is intelligence not his intelligence actually the right measurement for what we’re trying to achieve.
Or maybe it’s that ASI inevitability claims to be a very general argument, when it’s a very specific argument (or vice versa)
The AI inevitability argument seems to be something like - are you so anthropocentric and unimaginative that you can’t see a world where human creations outdo humans.
When the specific claim is more like “The specific technology of large language models will hard takeoff in the next few years”
There are infinite years ahead of us. Maybe hard takeoff is coming! And maybe it’s a thousand years away.
Maybe I’m not in the groups if people who overuse the word stochastic terrorism, but I can never find myself completely against the idea, since there’s a nonprofit that set up with the explicit purpose of helping parents whose children were killed in a school shooting prepare for the death threats they would get from listeners of Alex Jones.
I think I learn a decent amount from AI so I’m legitimately torn on this, but I guess an analogy I might turn to would be something like “fork lifts are great! But don’t bring them to the gym to do your heavy lifting.”
Cara are great! But a one hour drive serves a different purpose from a one hour run. Running doesn’t make me a Luddite.
So I think - it will continue to be important to “do the reps” so to speaks. Build mental models. Read long form. Do crossword puzzles. Think on your feet and have in person conversations.
I think / if a university carries the responsibility of credentialing you, they equally carry the responsibility of ensuring you didn’t bring a fork lift to the gym.
I was accused of being AI a few posts ago. I think it’s hard get the cadence out of your head if you talk to AI a lot. Especially when making certain kinds of arguments. I was thinking about dropping a few more f bombs, maybe that would seem less AI like.
I think it’s a maturity model question, and an individual ability question. And maybe an industry tooling question.
Individual ability? Newer engineers should read and write more code than experienced engineers.
Maturity model? Teams with better access to objective specifications can read less code than teams without it. Teams with more experience with building loops and validation frameworks the same.
Industry tooling? I think we need more work on the spec driven development front. Some set of standards as to how we define contracts that LLMs can operate against that are more deterministic.
In the context of many other measures I think speed is an important measure. Maybe even the key measure. I’ve even written a blog post entitled “reading code is an anti-pattern.”
But if the metric is, slowing us down from having more lines in production then the answer is unequivocally yes.
If the answer is providing customer value, or having a sustainable engineering culture. Then I’m not so sure. I’m not saying I believe the opposite, but it feels like excessively optimizing on the wrong thing.
Ultimately I think this asks the wrong question, I think most other surrounding questions is the right one which is - how do you safely and quickly deploy the right code to production that delivers customer value.
I think that will continue going forward involve doing so in an automated fashion, but then the right question isn’t, “should we stop reading code”. But something like “what is the right way to ship intent to production”
The whole LLM development revolution has made me feel like I’m taking crazy pills, and this is one of those observations.
And, as someone who does a decent amount of vibe coding -
Are we really turning engineering into plain text and not a like agreeably defined spec? Is that how we’re writing applications now?
That a list of instructions that may not have an agreed upon definition or compilation process to ensure they’re internally consistent - that’s the foundation of modern change management?
It's not the usage of AI so much as a seeming enthusiasm that this generation of tools are necessarily the right ones.
It's kinda weird because I think this almost loops back on itself, but a fundamental question is like - what is the difference between AI and a library?
What's the difference between typing say "I want to reach out to the google API and get a json blob of my e-mail" and "axios.get(google.com/e-mail)"
What's the difference between "Build me a login screen" and having a good set of react components you can quickly compose into a login screen.
As sort of illustrated in the article. It seems more reliable and potentially less expensive to have a layer that can sort of say "This line here, it's covered by this library/git hook, we'll actually pull it out, replace it with a deterministic layer and maybe put it in a spec that's ignored or verified by LLMs to have the deterministic component, but not interpreted by LLMs"
I don’t entirely disagree with this criticism, but it also feels to me like - progressivism is the flavor of modern bad studio films, but not a cause of bad media.
Like going back to the early 90’s there was this incredible rash of bad films, especially action movies, and that was just the nature of Hollywood incentives.
Because I think the complaint in some sense isn’t the fact that there are progressive movies, it’s the fact that it’s a layer added to all low risk big budget studio films.
But these movies would likely be bad for other identifiable reasons if not for the sprinkling of diversity. The lord of the rings on Amazon isn’t otherwise great.
So sometimes I think this criticism is just feeding into the very thing it’s trying to fight. Industry is great, and it’s great for the reasons described, but I think the moral ambiguity problem as described isn’t caused by diversity, they’re correlated. Bad movies have less moral ambiguity and use diversity in a hamfisted way.
There’s something of a footnote in here that - raises a moral or even spiritual question.
Scott says that one of the great things about superforcasters is that some people who regularly make bad decisions will be able to make better decisions by asking superforcasters questions like “Is this marriage likely to end in divorce.”
And there’s a number of profound questions in here about - what are we all here for?
There’s something of a philosophical horseshoe theory here which is like - I might argue - this sounds like astrology or other metaphysical forms of divination.
And the counter argument would be something like - no it’s completely different because this is true. Which is of course what all the spiritual traditions say about each other.
So it begs the question - is it the solution or the process.
Is using a super forecaster better because it’s “real”; or like I think I could make the argument it’s worse for that reason. Asking your grandmother or spiritual advisor is part of a process of existence. You ask multiple people, you have the experience, maybe you marry the wrong person, you learn and you grow.
> Why can't we have a red-queen's race where non-tech companies are implementing AI, but it's not increasing the total profits of those sectors, just meeting rising customer demands/fighting over the share of existing profits?
But then if this happens - all of the stock market has risen in the promise of AI. If AI eats profits instead of grows them, then the economy shrinks right? So maybe that’s worse? That there is no productivity increase?
> How much do these models need to do before people throw their hands in the air and say, ok this is happening.
It's quite interesting to me that your epistemology is basically "God in a box is inevitable" and not skepticism.
> Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).
The blocker is lots of things, but for one? Compute. Millions of instances running parallel, all "probabilistic" testing different theories would require million of GPU clusters.
Two? For subjects like Physics - Atoms. You need to be able to run experiments. You need access to the real world.
Then there's the S curve. Algorithmic breakthroughs operate on orders of magnitude. Eventually we'll run out.
I mean he explicitly at the end says there will be a new era of prediction markets:
"This, then, is my prediction for the AI superforecaster future: for basic questions, your off-the-shelf AI chatbot will be able to offer opinionated probabilities superior to those of any human. For more controversial or bias-laden questions, a new era of prediction markets will smooth over differences in brand and model and efficiently aggregate all AIs’ opinions."
But one reason that you would publish it for everyone else to react was exactly what I said, because if you can show people that a superhuman AI believes a certain outcome is more likely, you impact whether or not that certain outcome is more likely. Which feels like a flaw in this version of the future. That in fact you could overcorrect and make the markets less efficient.
I suppose on the one hand, I probably understand what this means?
But I hate when guidance is totally verboten because like -
“Babies should not be shaken at all in the first two years of life”. I think has a _significantly_ different impact meaning then when you say it about screens. I don’t think if my baby happens to glance over at the TV there will be significant impacts to health and development.
And I know that, but when you say it’s just verboten, it doesn’t actually provide any signal, and it’s similarly used for things that are less easily differentiated.
"You can absolutely write bad Pulumi code. You can over-engineer it. You can create abstractions nobody understands. You can make something simple feel clever for no reason.
But that is not a Pulumi problem.
That is a software design problem."
An uncomfortable idea is that these are basically the same thing. There's a reason the phrase "just enough rope to hang yourself" exists. A system that requires more software design <can be> a problem in and of itself.
Right, but if your real assertion is “we have no idea”, it seems you should point your skepticism significantly more towards the people betting $100 billion dollars that self-driving cars are coming next year than the ones who aren’t.
I don’t understand the analogy you’re making, or maybe I think it’s wrong. This is the first time ive ever seen someone say you should outsource your thinking to an LLM, rather than say idea generation.
No one thinks you shouldn’t do 8 digit multiplication with a calculator, But you should understand what it’s doing under thr hood so if you say typo something you can catch when the answer is off by an order of magnitude.
But the same argument applies to AI. If you don’t understand the basics of an argument or the nature of the subject you’re investigating, you can’t tell - not even an if it’s working correctly but if it’s responding to the question you asked. If it applied the right context for your particular situation.
And I think it’s the exact same thing - whether AI is really thinking is irrelevant, students need to understand the nature of how to make arguments and validate information, before they can trust their own usage of AI.
How is EFF asking to police or restrict users use of technology? The only thing they seem to be asking here is for X to continue to generate reports in how they use (or misuse) PII.
The concerns on Grok seem pretty specific: to not take for granted that it doesn’t introduce problems with how Twitter handles user data, not what users can do with it.
From the article:
“These sweeping assurances that corporate restructuring led to a fundamental change in X’s policy and practices around user data should be met with a healthy dose of skepticism, given evidence to the contrary. For example, the company’s quiet rollout integrated its AI model Grok with the platform in 2024, trained (without meaningful consent) on X user data. The company was also subject to a massive data breach in 2025.”
Nothing about groks capabilities or what users are allowed to ask it.
But I imagine there’s this problem that measuring intelligence in AI is not the same as measuring intelligence in humans. How is fable not already super intelligence. How is it not way smarter than the 99th percentile human? In terms of speed, broad ability and pattern matching behavior. If intelligence is loosely correlated with lifetime income, Anthropic billions in revenue should be a decent measure that LLMs are way smarter than most folks.
Because sometimes I think the inconvenient truth is that when people ask, you don’t think we’ll have AI that’s smarter than humans. It feels to me like we had AI that was smarter than humans a long time ago that somewhere around GPT three or GPT four we had AGI and if we already have AGI and we already have AI that’s smarter than humans then maybe the real question is is intelligence not his intelligence actually the right measurement for what we’re trying to achieve.
Or maybe it’s that ASI inevitability claims to be a very general argument, when it’s a very specific argument (or vice versa)
The AI inevitability argument seems to be something like - are you so anthropocentric and unimaginative that you can’t see a world where human creations outdo humans.
When the specific claim is more like “The specific technology of large language models will hard takeoff in the next few years”
There are infinite years ahead of us. Maybe hard takeoff is coming! And maybe it’s a thousand years away.