AI might not recursively self improve (part 2)(secondsight.dev)

AI might not recursively self improve (part 2)

https://secondsight.dev/2025/07/ai-might-not-recursively-self-improve-part-2/

3 comments

So, nothing about my mental model of the world thinks that AI recursively self-improving (limitlessly) is possible but wouldn't a basic sort of math-y version of this be that most improvements AI can find in itself would be linear, but the marginal cost of improvements would be exponential?

I get that none of this stuff is like solid, but recursive self-improvement feels similar to a perpetual motion machine.

Base assumptions about the fundamental nature of the world have changed in the past (I like some of the examples in `sapiens`, even arguing that the idea of being able to improve our environment as a species was not believed for most of history) so I could be wrong, but fast takeoff feels like a fantasy.

I also think about cookie clicker math. A new technological paradigm may give us exponential improvement, but we eventually hit the next exponential. i.e. when you buy a big upgrade in cookie clicker, it can start making your current exponential (say if you're at millions of cookies) really fast, the wheel spins really fast until you hit about 1 billion in which case the wheel slows down to a crawl.

ffwd작년

Yeah interesting I have to think about this.

darad작년

technically its not possible to have AI limitlessly improve just because nothing can hold infinity for to hold infinity you yourself have to be infinite(storage and data)