I disagree. In fact, I think the field of dev ops gives a clean analogy with mathematical proofs. My point was that my work often requires that I figure out a consistent way to prove to myself what the condition of a system is by asking the right probing questions about it. What I've seen is that even the best LLMs lack a good intuition about what questions they should be asking and instead reach for the quickest and most obvious checks that leave edge cases uncovered. Maybe it is something about the domain of the problem; I don't know. But it makes it hard for me to imagine that an LLM wouldn't make similar errors in other cases, especially when generating mathematical proofs that will soon be too dense for humans to review.
Exactly. I really wish people wouldn't use this. If this becomes popular, anthropic will just modify their cache policy to be much less fair. It's not like they have infinite cache.
Without any more context, "keep going" seems to be doing a lot of work. The user is placing a lot of faith in the LLM to not make subtle logic mistakes and to take good approaches to each problem. In my experience, even frontier models (such as Fable) are quite capable of getting confused during even simple technical work I've done in the dev ops world. For example:
LLM: This package hasn't made it to production.
ME: are you sure? i see it right here!
LLM: You're right to push back. I inferred that based on weak data. I see now that the package has been deployed!
If the above conversation is typical for me, how could one expect to achieve a sound result by repeatedly prompting an LLM to simply "keep going" in dense mathematical proofs? Perhaps the user in this case had actually checked the LLM's work before issuing the prompt, but I think you see my point anyway.
Also known as, the tu quoque fallacy. Just because politicians in both parties have been doing this for decades doesn't mean that this administration is not especially hypocritical for doing it after whinging so much about free speech and free markets.
I'm finding it extremely hard not to have a cynical perspective on all of this. There's an idea that I've been mapping onto this whole this, which could be called something like effective knowledge. Regular old knowledge is just information, or access to information. Effective knowledge is the integration of all that information into an understanding that can be acted upon. That requires things like time, money, and involves the usual socioeconomic hurdles that have separated people into groups like "laborers" and "knowledge workers". Sure, in theory "anyone" could read textbooks and learn, but only a select few have the time, money, mentors in their lives, and so forth to really do that.
The rise in capability of LLMs over the past year has basically removed a lot of these boundaries for people. Learning, building, and experimenting is a lot easier when you have a capable partner like Claude to help you along the way. Claude doesn't always get everything right, and you have to be a skeptic, but it's a lot better than nothing.
When I see the government restricting access to LLMs (or Anthropic as they were doing with Mythos before the whole Fable debacle), I basically just see the same old pattern of the ruling class moving to protect their advantage by keeping the great masses in ignorance. Broadening access to LLMs (i.e. effective knowledge) would put everyone on a more level playing field. But we can't have that, because politics, nations, the economy, blah blah reasons reasons. Guess utopia will just have to wait a bit longer.
But then again, this feels a lot like cryptography export controls. Those controls are in place, but I doubt anyone really thinks they work or make much of a difference. Software is not like nuclear weapons, and a data center is a much smaller lift than a Uranium enrichment facility. So maybe this is just a temporary roadblock. But let me tell you, I sure am ready for it to feel more like the government is working for (not against) the people.
Kind of highlights how ridiculous their notion of safety is in this case. By this measure, I guess making the model "safe" means making it play dumb and intentionally ignore security bugs that it notices in the code? And what will the eventual legality of this look like? "Yes, your honor, we allege that this AI system that was sold to us willingly and knowingly ignored a critical security vulnerability in our software system, thereby leading us to be hacked and causing our business to fold."
It's exactly the same problem as backdoors in crypto systems. Criminals will find the crypto that isn't broken and use it regardless (or make it for themselves), while the rest of us losers are stuck with the broken version that we're allowed to use.
On this issue of cyber security, it seems better if authorities just start acting like the cat is out of the bag instead of pretending like it isn't. ASI is basically here now, so what are we going to do about it? Let's not bother pretending otherwise.
On another note, I doubt this was anything other than a vindictive administration enacting revenge on a party that refused them. We all know the Trump admin's priorities.
I like to think that the long arc of history bends towards greater access to knowledge and intelligence. I mean, isn't that what we all want? To be collectively less ignorant and more aware of how the world works? But I guess that's not what the US gov wants. Crazy times, truly. The mask is really coming off lately.
I'm not sure what the mechanism is, but I've definitely had Claude refuse to work on sessions that were touched by other models. Some kind of integrity check failure. Resetting the session back to the point before I used the other model fixed the problem.
What I would buy: a local AI focused laptop with a built-in, powerful TPU. And it would have to open its hardware interface so that I could actually do what I wanted to do with it.