Never mind debates about intelligence. How many actions can a system take at 99% reliability before a human must intervene? This metric is doubling every 7 months.
Certainly one of my criteria for a utopia would be that some of the beings in it are engaged in intriguing or exciting activities with meaningful consequences. I'd require more than Special Circumstances to meet this criterion -- they are too small a piece of the total system.
In Player of Games we see a corner of a gaming culture which partly meets this criterion but it does not have meaningful consequences outside the gaming participants (unless you count the ways the Minds use it to manipulate Gurgeh).
Maybe by this criterion utopias are impossible, since the disruption caused by exciting activities with consequences conflict too much with the optimality of the society. But I don't think anyone now can prove this would be the case.
In the Culture Series history hasn't ended, and the members of the Culture can and do leave and live in ways the Culture would find repugnant. Also there are marginal participants (such as the Mind known as Meat Fucker) who are more or less in the Culture but act in ways the Culture finds repugnant, but are not punished or controlled.
So according to your definition the Culture is not a utopia.
This is an example that supports Scott's point that people don't have world models. The people who "believe" this don't wonder how stock market continues to operate now that NYC is a wreck. Etc.
I wonder in what sense they really do "believe". If they had a strong practical reason to go to a big city, what would they do?
> it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
This seems very incorrect, or at least drastically underspecified. These trains of thought are "normal" (i.e. common and unremarkable) so why don't they "latch your brain into a very undesirable state" lots of the time?
I don't think Scott or anyone up to speed on modern neuroscience would deny the coupling of mental state and brain chemistry--in fact I think it would be more accurate to say both of them are aspects of the dynamics of the brain.
But this doesn't imply that "simple normal trains of thought" can latch our brain dynamics into bad states -- i.e. in dynamics language move us into a undesirable attractor. That would require a very problematic fragility in our normal self-regulation of brain dynamics.
"we replicate the DeepSeek-R1-Zero and DeepSeek-R1 training on small models with limited data. We show that long Chain-of-Thought (CoT) and self-reflection can emerge on a 7B model with only 8K MATH examples, and we achieve surprisingly strong results on complex mathematical reasoning. Importantly, we fully open-source our training code and details to the community to inspire more works on reasoning."
They replicate the DeepSeek-R1-Zero and DeepSeek-R1 training on small models with limited data. We show that long Chain-of-Thought (CoT) and self-reflection can emerge on a 7B model with only 8K MATH examples, and they achieve surprisingly strong results on complex mathematical reasoning. They fully open-source their training code and details to the community to inspire more works on reasoning.
That matches how philosophers typically talk about the Chinese room. However the Chinese room is supposed to "behaves as if it understands Chinese" and can engage in a conversation (let us assume via text). To do this the room must "remember" previously mentioned facts, people, etc. Furthermore it must line up ambiguous references correctly (both in reading and writing).
As we now know from more than 60 years of good old fashioned AI efforts, plus recent learning based AI, this CAN be done using computers but CANNOT be done using just ordinary if - then - else type rules no matter how complicated. Searle wrote before we had any systems that could actually (behave as if they) understood language and could converse like humans, so he can be forgiven for failing to understand this.
Now that we do know how to build these systems, we can still imagine a Chinese room. The little guy in the room will still be "following pre-written scripted algorithmic procedures." He'll have archives of billions of weights for his "dictionary". He will have to translate each character he "reads" into one or more vectors of hundreds or thousands of numbers, perform billions of matrix multiplies on the results, and translate the output of the calculations -- more vectors -- into characters to reply. (We may come up with something better, but the brain can clearly do something very much like this.)
Of course this will take the guy hundreds or thousands of years from "reading" some Chinese to "writing" a reply. Realistically if we use error correcting codes to handle his inevitable mistakes that will increase the time greatly.
Implication: Once we expand our image of the Chinese room enough to actually fulfill Searle's requirements, I can no longer imagine the actual system concretely, and I'm not convinced that the ROOM ITSELF "doesn't have a mind" that somehow emerges from the interaction of all these vectors and weights.
Too bad Searle is dead, I'd love to have his reply to this.
Also, you are obviously wrong (or rhetorical?) when you say "No one is talking about what a machine intends to do." We certainly do! You can say "No one should" or other normative condemnations but then we're arguing on different territory.
Some philosophers take that position. Dennett, explicitly cited in the article, wrote The Intentional Stance (1987) about exactly the approach to intentionality taken in the article. His approach is accepted by many philosophers.
As you point out, the approach you cite can't be used in a materialist metaphysical position. That's a pretty severe problem for that definition! So Dennett's approach, or something like it, has major advantages.
Cool way of putting it. Let's run with that. A good actor can be seen as instantiating a Boltzmann Brain while on stage -- especially when improvising (as always may be needed). Maybe each of us is instantiating some superposition of Boltzmann Brains in everyday life as we wend our way through various social roles...
From now on I'll listen for the subtle popping sounds as these BBs get instantiated and de-instantiated all around me...
Of course a philosopher can object that (1) these BBs are on a substrate that's richer than they are so aren't "really" BBs and (2) they often leave traces that are available to them in later instantiations which again classical BBs can't. So maybe make up another name -- but a great way to think.
The article provides a very clear reason for using the idea of "intention": that framing helps us understand and predict the behavior. In contrast framing a mathematical function as having "intention" doesn't help. The underlying mechanism isn't relevant to this criterion.
Clearly the system we're understanding as "intentional" has state; we can engage in multi-round interactions. It doesn't matter that we can separate the mutable state from the function that updates that state.