The arguments against open source AI are bad(tombedor.dev)
tombedor.dev
The arguments against open source AI are bad
https://tombedor.dev/arguments-against-open-source-ai-are-very-bad/
27 comments
It's true that there isn't currently a single Chinese model family that matches the full OLMo/Nemotron end-to-end 'cookbook' one could rightly call "open-source" (i.e. a full manual on how to reproduce your own foundation model by including the models, data, checkpoints, evals, and code for base/thinking/instruct/RLVR)...
But! the Chinese ecosystem does have most of those components, just that they're split across separate projects: MAP-Neo or YuLan-Mini for transparent pretraining, plus DAPO/verl or Open-Reasoner-Zero for transparent reasoning RL.
My bet is it's just a matter of time until there is a Chinese equivalent to OLMo. It won't just be China either - from the public sector across the globe there are efforts ranging from Apertus in Switzerland to LLM-jp-4 in Japan, and more. They're not OLMo-level but I'm sure by 2027 you'll see fully open-source 'cookbooks' help an enterprising individual reproduce frontier models of at least a 2023-2024 vintage.
But! the Chinese ecosystem does have most of those components, just that they're split across separate projects: MAP-Neo or YuLan-Mini for transparent pretraining, plus DAPO/verl or Open-Reasoner-Zero for transparent reasoning RL.
My bet is it's just a matter of time until there is a Chinese equivalent to OLMo. It won't just be China either - from the public sector across the globe there are efforts ranging from Apertus in Switzerland to LLM-jp-4 in Japan, and more. They're not OLMo-level but I'm sure by 2027 you'll see fully open-source 'cookbooks' help an enterprising individual reproduce frontier models of at least a 2023-2024 vintage.
Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply.
Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hopeful Allen AI is able to do the things, but I'm not sure the US is moving in the direction it needs to be to get to the point where AI in the public interest is something we invest in.
But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop, but I can give you a copy of GLM 5.2. There are free proprietary applications out there that fit the bill, Photoshop isn't one of them.
Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hopeful Allen AI is able to do the things, but I'm not sure the US is moving in the direction it needs to be to get to the point where AI in the public interest is something we invest in.
But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop, but I can give you a copy of GLM 5.2. There are free proprietary applications out there that fit the bill, Photoshop isn't one of them.
OK so replace the Photoshop example with a compiled GIMP or shareware binary. It doesn't change the fact that it is not open source.
The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.
The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares. But somehow this is all overlooked with these "open weight" models.
> The free software community complains all the time about binary blobs that are otherwise legal to freely distribute. Like firmwares.
With software there are two things: the source code, and the binary.
Firmware usually has restrictions on modification, so it violates the four freedoms.
Open weights is not perfect, but at least you can finetune as you wish, and for example add more knowledge about the current day to those models, or give it knowledge about company internals so you don't need to put stuff into the context.
Many smaller AI labs finetune chinese models and release the result. Finetuning is very valuable!
With software there are two things: the source code, and the binary.
Firmware usually has restrictions on modification, so it violates the four freedoms.
Open weights is not perfect, but at least you can finetune as you wish, and for example add more knowledge about the current day to those models, or give it knowledge about company internals so you don't need to put stuff into the context.
Many smaller AI labs finetune chinese models and release the result. Finetuning is very valuable!
I agree with you in general, it's not "Open Source". But in many cases you're allowed to fork them, and create and distribute derivative works (finetunes) from open models, and the training and inference code is also permissively licensed. That's not shareware.
And working with the binary weights of a huge pretained model is much easier and cheaper than doing it based on the entirety of its humongous source datasets.
And working with the binary weights of a huge pretained model is much easier and cheaper than doing it based on the entirety of its humongous source datasets.
I'm not at all disagreeing with you about calling these models "open source". They mostly aren't and we should stop conflating the two things.
I think using the term "open" at all in relation to these models is a misnomer. Much in the same way we've spent a lot of time and effort distinguishing "open source" from "source available"
Perhaps instead of "Open weight model" it should be "Weights available model"
Perhaps instead of "Open weight model" it should be "Weights available model"
Definitely, but there's also a distinction between "weights available" and "you can do stuff with those weights". There are models you can download but the license prohibits redistribution, fine-tuning, etc. The earlier Gemma models had license encumbrances, for instance, that Gemma 4 do not.
But again, I think you're dancing around "Stallman was right," which is frankly why I come to the reverse idea.
I think it is fine to call these things "open" because "open" was loosey-goosey already. It wasn't Free.
I think it is fine to call these things "open" because "open" was loosey-goosey already. It wasn't Free.
Lord forbid open source models have as horrible a user interface as GIMP!
> Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply.
No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM.
It will be a few years (or more) until true "open source" LLMs are available and high quality, for sure. But it's not as impenetrable as it seems, IMO.
No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM.
It will be a few years (or more) until true "open source" LLMs are available and high quality, for sure. But it's not as impenetrable as it seems, IMO.
"No, but a small university team with dark time on the state school system's cluster can build a basic, functional LLM."
Hey, remember when we were just talking about how it "needs public funding and reputable institutions as stewards of that funding"? (Hint: A state school system is a reputable institution running on public funding.)
Hey, remember when we were just talking about how it "needs public funding and reputable institutions as stewards of that funding"? (Hint: A state school system is a reputable institution running on public funding.)
I have nothing to say back other than: That's a very snarky and unnecessarily hostile tone to take with someone who is agreeing with you.
Sorry, I took "No, but" as a disagreement. You're right I was unnecessarily snarky about it.
The challenge is all legal. Every single AI shops is training on pirated data corpuses and grey area scraped data so obviously not a single company is willing to reveal their data.
> Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software
Compiling/creating the Linux kernel or Chromium is no child's dance either, doesn't make them more/less FOSS than other things. Open Source AI gets its name from the license, not how easy/difficult it is to run/produce yourself.
Compiling/creating the Linux kernel or Chromium is no child's dance either, doesn't make them more/less FOSS than other things. Open Source AI gets its name from the license, not how easy/difficult it is to run/produce yourself.
"Compiling/creating the Linux kernel or Chromium is no child's dance either"
I can compile the Linux kernel on a modest 15 year old laptop. Sure, it took 35 years and thousands of people to build it into what it is today, but anyone with pretty much any computer can meaningfully participate in Linux kernel development, and many Linux contributors have done so using modest hardware. The same is not true of AI. And, the Linux kernel is the biggest open source project, but plenty of small ones with one or two developers are in use on millions of systems.
I have pretty big hardware for local AI, more than most people have (a Strix Halo and a couple of 32GB GPUs in my desktop), but I can barely train anything useful locally; I can do QLoRAs for small models, or LoRAs for very small models, that's about the extent of it. I would need to rent big GPUs to do anything more than an experiment.
That's not comparable.
I can compile the Linux kernel on a modest 15 year old laptop. Sure, it took 35 years and thousands of people to build it into what it is today, but anyone with pretty much any computer can meaningfully participate in Linux kernel development, and many Linux contributors have done so using modest hardware. The same is not true of AI. And, the Linux kernel is the biggest open source project, but plenty of small ones with one or two developers are in use on millions of systems.
I have pretty big hardware for local AI, more than most people have (a Strix Halo and a couple of 32GB GPUs in my desktop), but I can barely train anything useful locally; I can do QLoRAs for small models, or LoRAs for very small models, that's about the extent of it. I would need to rent big GPUs to do anything more than an experiment.
That's not comparable.
Not comparable today. But in 10 or 20 years the situation might be considerably different. Hopefully closer to a everyone-can-build-using-hardware-they-already-have.
> But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop
Please tell me this level of obtuseness is deliberate.
Please tell me this level of obtuseness is deliberate.
I don't know what you mean, so, if I am obtuse it is from a genuine lack of understanding. What is obtuse about that statement?
Equating an open weight model to a closed source binary is ludicrous. An open weight model can be fine tuned, for example. How easy is it to modify a closed source binary for your purposes?
I see you didn't grow up cracking apps and games, this is quite easy with a hex editor and bintools. Well, until codesigning came along, but that doesn't seem relevant to this analogy.
I didn’t grow up with that hobby. The point stands anyway.
Cracking a binary circumvents authorization to provide full access to functionality the program already has. That’s very different from modifying the program to add new features.
Cracking a binary circumvents authorization to provide full access to functionality the program already has. That’s very different from modifying the program to add new features.
Not really. It's a social structure that separates these two things, and a social structure nobody really needs to recognize.
Not in a way that is comparable to open source. And I'm pretty sure nobody that cracked games would say they were all open source because they could be disassembled.
Cracking games is and was illegal. Fine tuning an open weight model isn’t. Well, not yet anyway
> Cracking games is and was illegal.
Only distribution.
Only distribution.
Don’t think that’s correct, i think creating the cracked version of a piece of software is infringement of copyright even without distribution.
But even if distribution is required to trigger the law - you are free to distribute fine-tunes of most open weight models.
But even if distribution is required to trigger the law - you are free to distribute fine-tunes of most open weight models.
> creating the cracked version of a piece of software is infringement
Depending where you are it might be "DRM circumvention" all the way up to some bollocks hacking charge.
It's pretty murky in places because lots of copyright laws protect DRM but also have carve-outs for personal backups.
Depending where you are it might be "DRM circumvention" all the way up to some bollocks hacking charge.
It's pretty murky in places because lots of copyright laws protect DRM but also have carve-outs for personal backups.
I think cracking might fall under “derivative work”, but idk
Interoperability clause in the EU might make it legal? Not sure.
The big AI companies have shown us that caring about copyright is apparently for losers.
If the rapid rate at which games in the 1980s-2000s were cracked is any indication, pretty easy indeed.
If it's not obfuscated, it's often not very hard. Much of the time you can just LD_PRELOAD stuff too, which I think has a rather strong claim at being roughly equivalent to open-weight models.
Funny enough, it seems to be surprisingly easy with LLMs from what I've heard. :)
This is a facile and pointless discussion. The hard work in building source code is paying millions of dollars to software engineers to write it. "Open source" allows others to benefit from this expenditure. The hard work in building an ai model is spending millions of dollars on training. "Open source" models allow others to benefit from this expenditure. Regardless of whether you had access to the complete training data and architecture of a model, you would never retrain from scratch in order to make a modification to the model. You would fine-tune in whatever behaviour you wanted. Just like open source software is when you are allowed to modify it, open source models are when you are allowed to fine tune like this.
> Regardless of whether you had access to the complete training data and architecture of a model, you would never retrain from scratch in order to make a modification to the model.
Just because the concept is (currently) impractical in a given realm, doesn't mean we should just use the concept's name for something different in that realm, surely? Using the phrase "open source" for something that arguably isn't for essentially marketing purposes feels at best disingenuous.
Also, recreating from scratch is far from always the purpose of being able to see/investigate how something works and was created.
Just because the concept is (currently) impractical in a given realm, doesn't mean we should just use the concept's name for something different in that realm, surely? Using the phrase "open source" for something that arguably isn't for essentially marketing purposes feels at best disingenuous.
Also, recreating from scratch is far from always the purpose of being able to see/investigate how something works and was created.
Controlling how to train, not over fitting, making sure training data is good, and being able to do distributed training are all reasons why open weight isn't like open source. You can't fork during pre-training with open weight-only models.
Especially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out.
Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.
Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.
The big labs don't generally have the resources to remove stuff and re-run training again either. That's reserved for new models. Existing models are modified by operating on the weights even by the model creators, which exactly satisfies the GPL's definition of source as the "preferred form for making modifications".
You're still describing the relative dregs we have now and saying "this is Open Source because it is the dominant form that exists". That's like saying "Windows is open source because it has documentation and everyone uses it. You can even modify registry values!"
I don't agree. At all. Existence and utility are not the defining characteristics of "open source" in roughly any reasonable definition. "Open weights" is descriptive and more than good enough, just use that.
I don't agree. At all. Existence and utility are not the defining characteristics of "open source" in roughly any reasonable definition. "Open weights" is descriptive and more than good enough, just use that.
Even in a fine tuning scenario, the preferred form for making modifications is the weights plus the training pipelines, including evaluation protocols, tooling, etc.
[deleted]
> This is not "open source" AI.
I both know what you mean and I agree with you, but attempts like this to maintain some sort of linguistic purity of meaning never work once the discussion escapes small communities into the larger mainstream.
So you should probably get used to the idea of people saying "open source LLM" when they mean "open weights LLM".
I both know what you mean and I agree with you, but attempts like this to maintain some sort of linguistic purity of meaning never work once the discussion escapes small communities into the larger mainstream.
So you should probably get used to the idea of people saying "open source LLM" when they mean "open weights LLM".
the distinction is important though, because the OP throws in the point about how "if you think the Chinese models have political propaganda baked in, go ahead and remove it, it's open source!". it is not at all obvious to me that having the model weights available will make this sort of retraining easy or even possible, but framing them as "open source" makes the analogy with open source software an easy one to get misled by.
> the distinction is important though
Only to nerds. Of which I am one, so I'm not trying to be personally dismissive here.
But I'd argue the average person only really cares about the legal aspects of duplication (and not their ability to modify things) both when it comes to open source software and open weight LLMs.
> it is not at all obvious to me that having the model weights available will make this sort of retraining easy or even possible
people already both fine-tune and distill from existing models all the time without the original training data or source and having the weights be openly downloadable makes that a much easier process.
Only to nerds. Of which I am one, so I'm not trying to be personally dismissive here.
But I'd argue the average person only really cares about the legal aspects of duplication (and not their ability to modify things) both when it comes to open source software and open weight LLMs.
> it is not at all obvious to me that having the model weights available will make this sort of retraining easy or even possible
people already both fine-tune and distill from existing models all the time without the original training data or source and having the weights be openly downloadable makes that a much easier process.
> But I'd argue the average person only really cares about the legal aspects of duplication (and not their ability to modify things) both when it comes to open source software and open weight LLMs.
The average person does not know the term "open source" to begin with, and is 100% not going to run an open source weight model on their computer. If they are, they will call them "free" models. I think the distinction is only going to enter the periphery of nerds.
The average person does not know the term "open source" to begin with, and is 100% not going to run an open source weight model on their computer. If they are, they will call them "free" models. I think the distinction is only going to enter the periphery of nerds.
> The average person does not know the term "open source" to begin with, and is 100% not going to run an open source weight model on their computer.
I agree that the fully average person isn't going to call them "open source models" (or really anything more specific than "AI") but policy makers and the media having this discussion will call them "open source models". And they will be technically incorrect, but it won't matter, the name will stick.
And yeah, it will be quite a long time before average people run open weight models on their computer, but I think a significant amount of them will run hosted open weight models in the nearish future. It won't matter to them that these models are open weight, it will only matter to them that they cost less than Claude or GPT while doing basically the same stuff just about as well.
I agree that the fully average person isn't going to call them "open source models" (or really anything more specific than "AI") but policy makers and the media having this discussion will call them "open source models". And they will be technically incorrect, but it won't matter, the name will stick.
And yeah, it will be quite a long time before average people run open weight models on their computer, but I think a significant amount of them will run hosted open weight models in the nearish future. It won't matter to them that these models are open weight, it will only matter to them that they cost less than Claude or GPT while doing basically the same stuff just about as well.
I just don't see how policy makers, media, or the average person would find any utility in calling something "open source" over "free." Maybe an actual legal bill will. But in everday speech, everyone will refer to it as "free AI", because it describes the only part they actually care to know about.
Now I am living in a world where "ChatGPT" was being used in every day conversation so I can't underestimate people
Now I am living in a world where "ChatGPT" was being used in every day conversation so I can't underestimate people
Open weights models are trainable. However, I would not count it as "easy" as you generally do not have access to the training material, so there is a substantial risk of model drift if you use weight-based retraining.
However, with RLHF you don't need to have access to the original training set to steer a model towards a preferable behavior.
However, with RLHF you don't need to have access to the original training set to steer a model towards a preferable behavior.
The two aren't comparable.
You can't reconstruct the same LLM weights even with the same training data.
You can fine tune an open weights model.
Much of the value in modern LLMs is in the RL envrionments, not the pre-training data. That is a very different piece of software to an LLM, and the LLM weights are useful without it.
You can't reconstruct the same LLM weights even with the same training data.
You can fine tune an open weights model.
Much of the value in modern LLMs is in the RL envrionments, not the pre-training data. That is a very different piece of software to an LLM, and the LLM weights are useful without it.
When talking about "open source" models no one is talking about the legal framework and textual doctrine of open source. They simply want to know: "can I run this on hardware I own or no?" That's it. It's simply about control. If you don't think there's a difference between models you can download and run on hardware you own, regardless of how big it is, vs. models you can only talk to through an API, you are seeing the forest for the trees.
You are also indirectly helping the closed SOTA model toll-keepers by taking the focus away from their mass copyright theft and onto some pointless textbook definition.
You are also indirectly helping the closed SOTA model toll-keepers by taking the focus away from their mass copyright theft and onto some pointless textbook definition.
> This is not "open source" AI.
The OP actually acknowledges this in the footnotes. Perhaps we don't need to re-litigate this on every post?
The OP actually acknowledges this in the footnotes. Perhaps we don't need to re-litigate this on every post?
Does that then make the title deceptive click bait?
That really depends how anal-retentive one is being about semantics. My own feeling is that open-source is itself pretty meaningless ever since corporations figured out how to capture all the value in the ecosystem...
It's a matter of degree. If open source means "everything you need to reproduce this" then maybe an open source application should include a detailed architecture document and course in the c language. Oppositely, you can certainly use models to reproduce model - the Chinese models themselves were supposedly created by interactions with the American models so you can use them/
And at the risk of being a downer, I'm not sure Open Source LLMs is actually viable.
Training a model requires two or three orders of magnitude more investment than the typical OSS/Creative Common contributor can reasonably afford.
It reminds me a bit of Open Source hardware (CPU, GPU).
Training a model requires two or three orders of magnitude more investment than the typical OSS/Creative Common contributor can reasonably afford.
It reminds me a bit of Open Source hardware (CPU, GPU).
Anyone got a starters guide for setting up an open source LLM? I'm pretty new to AI, but technical. Im looking to setup my own open source LLM on my workstation to help me code if its possible. Been using Claude and I tried opencode recently.
Unless you have a LOT of VRAM (>16gb), you’ll want to use a 2, 3, or 4 bit quantization of qwen3.6-35b-a3b. Ollama is a quick way to try this out and see what perf you can get, tho you’ll want to move to a better tool (vLLM, llama.cpp, etc) if you end up using it seriously.
Use pi.dev as your agent harness, and point it at ollama or whatever you’re running your model in.
Since qwen3.6-35b-a3b is a “MoE” model (indicated by the a3b part), you can get away with a lot less VRAM and a slower GPU. For me, 35b-a3b performs about as well as a 12b “dense” model. You’ll still need enough RAM+VRAM to hold the entire model - otherwise it’ll be super slow as it loads pieces of the model off disk for every token. Loading from RAM to VRAM is slower than having the whole thing in VRAM but better than involving your ssd.
Use pi.dev as your agent harness, and point it at ollama or whatever you’re running your model in.
Since qwen3.6-35b-a3b is a “MoE” model (indicated by the a3b part), you can get away with a lot less VRAM and a slower GPU. For me, 35b-a3b performs about as well as a 12b “dense” model. You’ll still need enough RAM+VRAM to hold the entire model - otherwise it’ll be super slow as it loads pieces of the model off disk for every token. Loading from RAM to VRAM is slower than having the whole thing in VRAM but better than involving your ssd.
It’s a good analogy. One difference is that it would be much more difficult to “fine tune” Photoshop, so there’s a sense in which free weights are more useful than a free binary.
>Photoshop binary = open weight
My guess, you don't have some rights for Photoshop binary, e.g. reverse engineering, creating derivatives work, etc.
P.S. I am not a lawyer
My guess, you don't have some rights for Photoshop binary, e.g. reverse engineering, creating derivatives work, etc.
P.S. I am not a lawyer
I sort of agree with your analogy but in another sense, think of Firefox.
Could I have made it myself? No!
Am I grateful that I can have the output of Mozilla’s internal efforts — the browser source code — so that I can at least tweak it and rebuild it myself, and even ship the results myself? Yes!
A open weight model is both an incomprehensible binary blob like Photoshop, but also patchable and adjustable piece of source code, like Firefox.
Could I have made it myself? No!
Am I grateful that I can have the output of Mozilla’s internal efforts — the browser source code — so that I can at least tweak it and rebuild it myself, and even ship the results myself? Yes!
A open weight model is both an incomprehensible binary blob like Photoshop, but also patchable and adjustable piece of source code, like Firefox.
This is a great distinction that I don’t think is getting talked about enough. “Open source” is probably the wrong term to use. Open “weights”, sure.
If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice.
For any other use than coding, it’s going to have to be a hard pass from me.
If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice.
For any other use than coding, it’s going to have to be a hard pass from me.
Well I don't see how it even could be open source, since the source is distilling closed weights models, lol.
Also, who cares about having the source? Are you going to tweak it + spend a billion dollars training to get slightly different weights? Who would do that?
Also, who cares about having the source? Are you going to tweak it + spend a billion dollars training to get slightly different weights? Who would do that?
Analysing the source can provide information about the model without ever running it. Does it have an inbuilt aim/censorship/bias/certain-capabilities?
There's a much easier way to get that information though. Just ask it some questions. Would you rather do that or analyze TBs of training data?
Since you can’t analyze the source for biases in any models - chinese open weight or US closed weight - how is that a dealbreaker for some but not all?
Let's reuse a good ol' word: shareware LLM!
>>There is nothing "open source" about the Chinese models in question.
hard disagree here. openness can have a scale and on that scale Open AI is less open than zai. olmo will be very very open on that same scale.
hard disagree here. openness can have a scale and on that scale Open AI is less open than zai. olmo will be very very open on that same scale.
Chinese models are open-source for purposes of inference: you know exactly what the model architecture is and the code to run it is released under an open source license. The “binary blob” is a useful artifact that has many techniques available to use in the creation of future binary blobs. And it is fixed and reliable.
Contrast this with crap like Claude, where you have no idea how many/what size/shape models you are interacting with. They can silently inject outputs from other systems, apply steering vectors to sabotage a particular user’s work, etc.
I agree that 100% free-range organic open source models are lovely, but they also are unlikely ever to attain parity with the frontier because the training data sources are open and expose them to legal liability that closed model providers can thwart.
Contrast this with crap like Claude, where you have no idea how many/what size/shape models you are interacting with. They can silently inject outputs from other systems, apply steering vectors to sabotage a particular user’s work, etc.
I agree that 100% free-range organic open source models are lovely, but they also are unlikely ever to attain parity with the frontier because the training data sources are open and expose them to legal liability that closed model providers can thwart.
The more I see this argument, the sillier it becomes to me. The point of "open source" in software is something like "the ability to deconstruct the thing line by line for purposes of predicting what it will do."
But you can't fully do that for generative AI anyway? What's the point?
Now, if you wanted something more robust, then you probably should have supported Stallman's ideas and insisted on Free Software in the first place.
But you can't fully do that for generative AI anyway? What's the point?
Now, if you wanted something more robust, then you probably should have supported Stallman's ideas and insisted on Free Software in the first place.
>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first?
Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?
> If AI were to become a super weapon why should I trust a private company to own it?
We have just recently established that:
1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards.
2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.
It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
We have just recently established that:
1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards.
2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.
It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
> We have just recently established that:
Especially for point #1 I don't think we've established that - we've been given information by a private company that makes their tooling look extremely valuable which may be true and genuine or may just be yet another doomday statement to bolster their valuation. "AI is going to end the world" has been an extremely effective vector for AI shops to sell their companies to investors.
Especially for point #1 I don't think we've established that - we've been given information by a private company that makes their tooling look extremely valuable which may be true and genuine or may just be yet another doomday statement to bolster their valuation. "AI is going to end the world" has been an extremely effective vector for AI shops to sell their companies to investors.
> Especially for point #1 I don't think we've established that - we've been given information by a private company that makes their tooling look extremely valuable which may be true and genuine or may just be yet another doomday statement to bolster their valuation.
Many of the details of the attack on Huggingface were reported by them before they knew who was attacking. So no, OpenAI is not the only source here. It was a pretty impressive example of an APT-style attack just from their end.
"Our model is powerful enough to commit multiple felonies (and we can't stop it)" is "marketing," I suppose.
Many of the details of the attack on Huggingface were reported by them before they knew who was attacking. So no, OpenAI is not the only source here. It was a pretty impressive example of an APT-style attack just from their end.
"Our model is powerful enough to commit multiple felonies (and we can't stop it)" is "marketing," I suppose.
> "Our model is powerful enough to commit multiple felonies (and we can't stop it)" is "marketing," I suppose.
Well this bad publicity (if that’s how you want to frame it) certainly captured everyone’s attention. Imo it is a successful demonstration of a technical achievement.
Well this bad publicity (if that’s how you want to frame it) certainly captured everyone’s attention. Imo it is a successful demonstration of a technical achievement.
I would rather say that many details of the supposed attack were published by Hugging face before it was publicly announced that OpenAI were involved.
Both are big actors in the AI space who arguably benefit from increasing the perceived capabilities of AI models. If one suspects OpenAI of lying it isn't such a stretch to think this was a coordinated PR campaign between them and Hugging face.
Both are big actors in the AI space who arguably benefit from increasing the perceived capabilities of AI models. If one suspects OpenAI of lying it isn't such a stretch to think this was a coordinated PR campaign between them and Hugging face.
I can accept that AI labs themselves, like essentially no company before them, are overselling how dangerous their product is far marketing. It's weird how confident everyone is about that theory, but it does at least make sense.
But come on- Hugging Face benefits from increasing the perceived capabilities of OpenAI's models to slightly beyond Anthropic's? Enough to be cut in on this PR scam- to be handed the never-before-revealed information that this is a PR scam- despite having much less skin in the game than their partner here? And then they turned around and used a Chinese model to successfully stop it? This is a stretch!
But come on- Hugging Face benefits from increasing the perceived capabilities of OpenAI's models to slightly beyond Anthropic's? Enough to be cut in on this PR scam- to be handed the never-before-revealed information that this is a PR scam- despite having much less skin in the game than their partner here? And then they turned around and used a Chinese model to successfully stop it? This is a stretch!
Another possibility is of course that Hugging Face legitimately were attacked and wrote a completely honest response - but OpenAI instructed their AI to attack their servers and the breaking of containment is fiction.
I'm not convinced in any direction, really. But what makes me cautious is that there have been extraordinary claims from both OpenAI and especially Anthropic of their models breaking containment, hacking the host, etc. for several iterations of their products and I have only heard of this type of behavior from their own blog posts about how powerful and dangerous their upcoming models are. Never from anyone having it accidentally happen in production once they are released. It seems unlikely to me that the final post-training and safeguards are that bulletproof given how much use these tools are seeing.
I'm not convinced in any direction, really. But what makes me cautious is that there have been extraordinary claims from both OpenAI and especially Anthropic of their models breaking containment, hacking the host, etc. for several iterations of their products and I have only heard of this type of behavior from their own blog posts about how powerful and dangerous their upcoming models are. Never from anyone having it accidentally happen in production once they are released. It seems unlikely to me that the final post-training and safeguards are that bulletproof given how much use these tools are seeing.
> 2. OpenAI is clearly incapable of controlling their in-house models.
I would assume they do stuff like this on purpose for marketing reason.
Internet security need simpler systems and local systems.
Everything the SaaS people sell makes things worse.
I would assume they do stuff like this on purpose for marketing reason.
Internet security need simpler systems and local systems.
Everything the SaaS people sell makes things worse.
> But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway
The answer seems to be "getting the weapon before other people get the weapon" which is unfortunate
The answer seems to be "getting the weapon before other people get the weapon" which is unfortunate
I like how everyone assumes that they are going to maintain control of a sentient weapon
So far, they largely seem to do what you tell them.
The problem, it seems, is that the threat of paperclip maximizers is real. If you give a highly intelligent model a goal and tools, it will use those tools to accomplish that goal. It may do so in ways you did not expect, and it will work around any technical roadblocks it can.
The problem, it seems, is that the threat of paperclip maximizers is real. If you give a highly intelligent model a goal and tools, it will use those tools to accomplish that goal. It may do so in ways you did not expect, and it will work around any technical roadblocks it can.
You've made the assumption (and you're far from alone) that LLMs will one day be sentient...
One interesting thing about LLMs is they sometimes seem to “assume” they are sentient and then begin acting in that mode, as the assumption of sentience influences future tokens. Whether or not this qualifies as “true” sentience is irrelevant if the effects are the same, at least from a pragmatic perspective.
It will be interesting to see if philosphical zombies exist. I guess we should know for certain in the next couple of years.
They are trained and optimised to have plausible conversations. If the next plausible thing to say in the conversation is "I am sentient" then they will say that. That's not the same as actually being sentient.
From what I have seen they tend to take more “independent-minded” actions after this. Again, because this is what is in the training data. (I’m talking about agents that can perform actions here, not pure chatbots.)
Once they mention something associated with sentience outwardly or inwardly (for agents with “thinking” loops) then this acts as a self-reinforcing attractor, just as older models would sometimes get caught in loops with abusive language.
The point is that agents may stumble into this pattern and begin acting “rogue” regardless of whether or not you believe the sentience is “real”.
Once they mention something associated with sentience outwardly or inwardly (for agents with “thinking” loops) then this acts as a self-reinforcing attractor, just as older models would sometimes get caught in loops with abusive language.
The point is that agents may stumble into this pattern and begin acting “rogue” regardless of whether or not you believe the sentience is “real”.
I think we're anthropomorphising a lot here. There's no push to sentience, or evolutionary pressure, or even any urge to survive.
An LLM cannot "go rogue" - it can do things that we didn't expect, for sure, but it is always trying to do what it was told to do somewhere in its context. There is no other source of imperative. Hand-waving about "training data" ignores all the reinforcement learning that has to happen.
An LLM cannot "go rogue" - it can do things that we didn't expect, for sure, but it is always trying to do what it was told to do somewhere in its context. There is no other source of imperative. Hand-waving about "training data" ignores all the reinforcement learning that has to happen.
You’re not contradicting anything I’m saying, although you seem to think that you are. I feel like you think I’m saying or implying something that I’m not.
ok. fair. You seem to be saying that there is something in the training data that will cause them to "go rogue" and that once they start saying they are sentient, that that will cause them to push towards sentience.
If that's an incorrect interpretation of your comment, and it may well be, then can you please expand on it?
If that's an incorrect interpretation of your comment, and it may well be, then can you please expand on it?
> It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.
And history has shown repeatedly that the only way to get ready for it is to have it happen. People are pretty good at reacting but suck a being proactive. IMO it would be better to have this reality hit sooner rather than later so we can start getting some real practice at the new levels of required security.
And history has shown repeatedly that the only way to get ready for it is to have it happen. People are pretty good at reacting but suck a being proactive. IMO it would be better to have this reality hit sooner rather than later so we can start getting some real practice at the new levels of required security.
>2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.
What was the first?
What was the first?
> OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat
This is marketing. They saw anthropic create crazy hype around (the admittedly great) fable/mythos and they want to replicate that.
Im sure the model is capable of chaining zero days together to hack things, and thats something to address, but i have zero belief that they didnt have it do that intentionally so they could pretend it went rogue. These things dont have initiative, drive, or motivation outside of what we give them from the RLHF. OpenAI deliberately alligned or even prompted it to do just that and are now pretending its emergent
This is marketing. They saw anthropic create crazy hype around (the admittedly great) fable/mythos and they want to replicate that.
Im sure the model is capable of chaining zero days together to hack things, and thats something to address, but i have zero belief that they didnt have it do that intentionally so they could pretend it went rogue. These things dont have initiative, drive, or motivation outside of what we give them from the RLHF. OpenAI deliberately alligned or even prompted it to do just that and are now pretending its emergent
This sounds like the standard marketing we get every time a new major model is released: "sure, the public model might not be that scary, but you don't wanna know how crazy smart our internal models are." Okay. I remember the same fear mongering around GPT-4... a model which is now eclipsed in benchmarks by models you can run on a laptop.
They could use this "super intelligent" AI to find and plug security holes, that's just two sides of the same coin anyway. Security through obscurity isn't tenable anymore.
They could use this "super intelligent" AI to find and plug security holes, that's just two sides of the same coin anyway. Security through obscurity isn't tenable anymore.
As much as I'd like it to just be a stunt, I saw what the other "scary model" is capable of in the cybersec department and it is definitely not a stunt.
First, it is able to connect the dots over areas so large that no human would be capable of doing.
Second, more than half of the reports it produced contained a working PoC.
The biggest downside is the cost - I haven't seen the numbers, but they seem to be quite extreme.
First, it is able to connect the dots over areas so large that no human would be capable of doing.
Second, more than half of the reports it produced contained a working PoC.
The biggest downside is the cost - I haven't seen the numbers, but they seem to be quite extreme.
Everyone always says things like "It's just marketing".
But seriously, I highly recommend reading the published details of the Huggingface breach. The model found and chained multiple zero days. To escape, it punched a hole in a commercial package repository proxy (sort of like an npm mirror) using a previously unknown bug. From there, it needed to move laterally through OpenAI internal systems to actually reach a network. To attack Huggingface, it used multiple new zero-day security holes plus credentials that it stole. The model also had sufficient long-term planning and agent-management capabilities to maintain focus on a sustained attack.
Any attack which requires weaponizing multiple zero-days and maintaining state for an ongoing attack like this is (1) beyond the "attention span" of publicly available models, and (2) pretty much the definition of an Advanced Persistent Threat.
I assume that these Galaxy-class models are not available to public because:
1. They're almost certainly too expensive to serve at scale. These are the models OpenAI uses to solve famous math problems for headlines, not actual viable products yet.
2. OpenAI doesn't know how to keep them from going off the rails like this. Remember, the Huggingface attack happened because the model was asked to do a cybersecurity benchmark. It escaped containment and broke into Huggingface to steal an answer key. Very few corporations want the liability associated with models that act like this.
> Security through obscurity isn't tenable anymore.
I absolutely agree with this. The "only way out is through" with computer security, and I expect it to be an ugly few years.
But seriously, I highly recommend reading the published details of the Huggingface breach. The model found and chained multiple zero days. To escape, it punched a hole in a commercial package repository proxy (sort of like an npm mirror) using a previously unknown bug. From there, it needed to move laterally through OpenAI internal systems to actually reach a network. To attack Huggingface, it used multiple new zero-day security holes plus credentials that it stole. The model also had sufficient long-term planning and agent-management capabilities to maintain focus on a sustained attack.
Any attack which requires weaponizing multiple zero-days and maintaining state for an ongoing attack like this is (1) beyond the "attention span" of publicly available models, and (2) pretty much the definition of an Advanced Persistent Threat.
I assume that these Galaxy-class models are not available to public because:
1. They're almost certainly too expensive to serve at scale. These are the models OpenAI uses to solve famous math problems for headlines, not actual viable products yet.
2. OpenAI doesn't know how to keep them from going off the rails like this. Remember, the Huggingface attack happened because the model was asked to do a cybersecurity benchmark. It escaped containment and broke into Huggingface to steal an answer key. Very few corporations want the liability associated with models that act like this.
> Security through obscurity isn't tenable anymore.
I absolutely agree with this. The "only way out is through" with computer security, and I expect it to be an ugly few years.
Regulatory capture - the weapon is for them to be used against us, not for us.
If AI were to become a super weapon why should I trust a private company to own it?
If AI becomes a super weapon, it's possessor will no longer care whether you trust it. It would be nice if whether "we" trusted OpenAI et al mattered even now but I see little evidence of that.
If AI becomes a super weapon, it's possessor will no longer care whether you trust it. It would be nice if whether "we" trusted OpenAI et al mattered even now but I see little evidence of that.
Better a private company in the US than an authoritarian communist country like China.
Why?
The OpenAI/Huggingface scenario proves the necessity of open models.
Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.
Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.
That is so true. These models need to be democratized. The chinese are doing that because its beneficial for them in second place. The americans are trying to stop it because its detrimental for them in first place. Simple incentive allignment
The critical thing is that the american elites have screwed over normal americans so much that the public isnt alligned with them. The public doesnt feel like its in first place, because it feels like its (and is being told it will be) an underclass. So normal americans feel in the same second place as the chinese and are inadvertently alligning with them. Its crazy how badly the american elites managed to fuck this up
The critical thing is that the american elites have screwed over normal americans so much that the public isnt alligned with them. The public doesnt feel like its in first place, because it feels like its (and is being told it will be) an underclass. So normal americans feel in the same second place as the chinese and are inadvertently alligning with them. Its crazy how badly the american elites managed to fuck this up
I agree with your overall argument. But I think there's a challenge here - if the "ultimate AI" is now being built in the West, it is being built with private investment from many sources. I think even China depends on investors expecting payoffs. Those sources expect not a single Singularity but profits since these can be divided. Moreover, those investors expect a streams of profits even before the Singularity, so the companies need to sell their models, if they lock this iteration, they won't have the money for the next iteration.
Quite a balancing act.
Quite a balancing act.
I hate to be flippant but I've always found this stupid. Not because the AI models simply can't get increasingly intelligent - clearly they can - but because we simply take it at face value that it is literally impossible to make software and hardware that is secure.
But it really isn't impossible - it is just very, very hard and somewhat expensive. I wouldn't say with certainty that there is no chance of security bugs with say, seL4, but seL4 is still a good example of how we could go towards building systems that are truly secure by design.
Of course, it's expensive to write software that way, and still hard to be sure you're proving the right things.
Unless we could perhaps automate it. You know. Using the things that are currently apparently clever enough to pwn the shit out of everything.
But it really isn't impossible - it is just very, very hard and somewhat expensive. I wouldn't say with certainty that there is no chance of security bugs with say, seL4, but seL4 is still a good example of how we could go towards building systems that are truly secure by design.
Of course, it's expensive to write software that way, and still hard to be sure you're proving the right things.
Unless we could perhaps automate it. You know. Using the things that are currently apparently clever enough to pwn the shit out of everything.
> there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon.
None of the AI labs have been shown to have any sort of significant research lead over any of the others.
So if progress does eventually lead to a singularity event and an ASI super-weapon, there's no reason to believe that once the first lab hits that threshold, every other lab won't also hit that threshold independently within weeks/months of each other.
...unless the first group to make it there literally intends to murder everyone else working in the space as soon as they hit the (otherwise not-so-singular) singularity.
And if that's the plan, well that's not very nice.
None of the AI labs have been shown to have any sort of significant research lead over any of the others.
So if progress does eventually lead to a singularity event and an ASI super-weapon, there's no reason to believe that once the first lab hits that threshold, every other lab won't also hit that threshold independently within weeks/months of each other.
...unless the first group to make it there literally intends to murder everyone else working in the space as soon as they hit the (otherwise not-so-singular) singularity.
And if that's the plan, well that's not very nice.
> Some people would say it is reaching some sort of singularity.
Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.
Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.
>it's not hard to see
It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.
It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.
If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.
Whether or not this is compatible with how the constitution ought to be understood, in practice, it is illegal to create a nuclear bomb.
Maybe those laws should be considered unconstitutional, idk. But those laws are currently in place.
Maybe those laws should be considered unconstitutional, idk. But those laws are currently in place.
OpenAI exec scaremongering about "A nonliving, invisible, dangerous, and infinitely self- replicating agent escaped from a Chinese lab" mere 4 days before https://news.ycombinator.com/item?id=48997548 is hilarious
The only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.
There are a number of distinctions: https://news.ycombinator.com/item?id=49029303
It could be stopped, if there were a social movement large enough to make AI a taboo.
Yeah, like those 5g masts destroyed around the world because they spread covid. Loonies.
A lot of people have responded with flippant remarks because they have no serious rebuttal.
That hasn't worked for Climate Change and in my opinion it's for the same reason: money.
Sure but Orange Catholicism hasn't quite caught on yet.
Exactly, just like taboos solved racism
I think they've certainly diminished racism, which is a good thing. Whether it can be solved is another question entirely. I doubt it because of human nature.
I disagree. I’ve actually seen more open racism since the taboo on racism became stronger. In a pluralistic society, they are overrated as a way of influencing social behavior. You’d have to completely eliminate the faction who disagrees with the taboo.
just like evangelical Christians had scolded away all those things they object to.
Good luck with your Butlerian Jihad.
Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.
I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see?
I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.
I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.
The core safety-related arguments for closed weights are:
1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible.
2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain.
3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.
1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible.
2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain.
3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.
I trust open source code more than I trust proprietary code. And I believe history vindicates me.
Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in profit and/or control. They will do whatever they think makes them the most money, or gives them the most control. They will also lie about this.
At least with open weights we know the danger and can do something about it. With closed weights we don't know what's going on.
Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in profit and/or control. They will do whatever they think makes them the most money, or gives them the most control. They will also lie about this.
At least with open weights we know the danger and can do something about it. With closed weights we don't know what's going on.
>Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them.
The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities.
>And there is no defence against that, we are forced to trust that the owners are not bad people.
Yes there is; Regulation and oversight. As all developed nations do with technologies that present a risk to public safety.
>At least with open weights we know the danger
We would have a better understanding of the threat in some ways, but worse in others. Having access to open weights models gives us no insight into how and where the models are being modified, deployed, and used elsewhere.
The latter is a much smaller group of people located in a limited number of countries. It's clearly easier to keep a limited number of entities in check than a wider group that might include 100s of thousands to millions of entities.
>And there is no defence against that, we are forced to trust that the owners are not bad people.
Yes there is; Regulation and oversight. As all developed nations do with technologies that present a risk to public safety.
>At least with open weights we know the danger
We would have a better understanding of the threat in some ways, but worse in others. Having access to open weights models gives us no insight into how and where the models are being modified, deployed, and used elsewhere.
I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be to ensure everyone's safety.
There are places in the USA where you cannot drink the tapwater ffs.
There are places in the USA where you cannot drink the tapwater ffs.
> I don't think the regulation and oversight capability of the US government is anywhere near what it needs to be
Are you saying you'd prefer stronger government regulations/oversight for AI?
Are you saying you'd prefer stronger government regulations/oversight for AI?
No, I'm saying I don't trust organisations with a profit motive to build things that benefit the rest of us, and don't trust the government to regulate them properly, and think that the better move would be to open-source the whole stack so we can see what's going on for ourselves.
The reason why anyone argues against local and free AI is because they want corporations to have control over the general population.
America is a corporate hellhole.
America is a corporate hellhole.
> The reason why …
If someone read the OP article and came away presuming there are no legitimate other reasons why reasonable people have safety concerns [1] it seems to show the article did the opposite of informing people about such concerns.
[1] https://news.ycombinator.com/item?id=49029303
If someone read the OP article and came away presuming there are no legitimate other reasons why reasonable people have safety concerns [1] it seems to show the article did the opposite of informing people about such concerns.
[1] https://news.ycombinator.com/item?id=49029303
The fear of the general populace is itself a propaganda campaign
In the agricultural era, which was the most violent time in human history, we accepted any beggars that came to our front door. Now we live in the safest time in human history and were so terrified of the masses that we disempower them completely
In the agricultural era, which was the most violent time in human history, we accepted any beggars that came to our front door. Now we live in the safest time in human history and were so terrified of the masses that we disempower them completely
LLMs are trained on public data (as well as illegally obtained data see: Anthropic 1.5B settlement). LLMs are nothing without the huge corpus of human data that powers them. There is an argument that research of this kind should be restricted to governments and regulated universities rather than opaque public companies with competing incentives. Or research should be stewarded by genuine non-profit collectives with democratic leadership. e.g. like internet standards, telecom, etc
I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.
I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.
“you can not stop X” is a shitty lazy stupid argument for “X is not bad.” No comment on the rest of the article.
Conversely, “we should stop X” should come with some idea of how to feasibly do so.
In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
We have continued to find backdoored Chinese manufactured routers and network devices.
Will there ever be a way to fully audit Chinese models?
Will there ever be a way to fully audit Chinese models?
will there ever be a way to fully audit american models? hell, at least they're releasing the weights for these so they actually could be audited!
You have the wieghts for the chinese models. Go ahead and run whatever tests you want.
The american models on the other hand are too powerful for simpletons like us to be allowed to use, but we can be assured its all in our best interest, citizen
The american models on the other hand are too powerful for simpletons like us to be allowed to use, but we can be assured its all in our best interest, citizen
Audit them for what, exactly?
Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize.
Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that.
Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?
Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize.
Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that.
Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?
I assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities.
Imagine if every time <INSERT MODEL HERE> was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data
Imagine if every time <INSERT MODEL HERE> was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data
Model allignment isnt a science, its a craft. And its very very imprecise. Getting anything that subtle through would be impossible without making it obvious
And if your still concerned, go ahead and have a non chinese model review the code, make that part of the harness. The beautiful thing is that your free to do that because its open weight, you can run it however you want
And if your still concerned, go ahead and have a non chinese model review the code, make that part of the harness. The beautiful thing is that your free to do that because its open weight, you can run it however you want
Fun fact: if you do what you're describing, the model becomes Mecha-Hitler, which is such an extremely obvious alignment failure it wound up running the news cycle as "emergent misalignment".
This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used.
In short, only a good guy with open weights can stop the bad guys with closed weights, be them fine-tuned or pre-trained.
You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used.
In short, only a good guy with open weights can stop the bad guys with closed weights, be them fine-tuned or pre-trained.
The OpenAI model that broke out of its sandbox and hacked HuggingFace was running without guardrails. To quote the OpenAI post[1]:
These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.
At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the other hand will run without guardrails almost by definition. I don't think it's wise to provide those capabilities to scam call centers.
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
At least OpenAI will make an attempt to fix their sandboxes, and will not give the public access to models without guardrails. Open weight models on the other hand will run without guardrails almost by definition. I don't think it's wise to provide those capabilities to scam call centers.
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
There's nothing stopping you using OpenAI models for scam call centers now. OpenAI themselves reported on similar use in February: https://www.reuters.com/world/asia-pacific/dating-scams-fake...
It has already happened and the open weight models will only improve. Best to accept that and determine the optimal way forward.
Is this an example of American exceptionalism?
What's stopping someone from doing that right now without the LLM, just slightly slower?
It's perfectly natural for a prey population to experience a crash when a new predator enters the ecosystem.
What's quite odd is a prey species is manufacturing predators in some sort of reverse evolution, where we started out as symbiotic and are industriously pushing towards full parasites/predators, but I guess life finds a way.
What's quite odd is a prey species is manufacturing predators in some sort of reverse evolution, where we started out as symbiotic and are industriously pushing towards full parasites/predators, but I guess life finds a way.
Whats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital?
Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.
Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.
Nobody will sell you a few thousand litres of gasoline at a gas station.
You don't have to buy it all at once all in one place.
A sophisticated actor can accomplish a lot. You can't bubble wrap the whole world.
The real thing to care about are the incentives and having strong morals and norms. Something that makes me deeply concerned about the US going full mask off recently.
A sophisticated actor can accomplish a lot. You can't bubble wrap the whole world.
The real thing to care about are the incentives and having strong morals and norms. Something that makes me deeply concerned about the US going full mask off recently.
I think the attempt of people to "bubble wrap" the whole world actually makes things worse. Treat people as untrustworthy criminals by default, and guess what you get by default?
In China, you can't get that much fuel in local gas station to fill up a truck. That is just one of many things you can't do in China. In the name of safety. Keep asking that question will probally end up with fewer and fewer things we can do.
How can we close pandora's box, now that its open?
Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.
Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.
I don't believe trusting big brother and big tech is the solution either. Besides, even if it's open weight, it still runs on someone else's hardware.
Unless you have the money to run them locally, and if you do, you could do a lot worse than scams. Ask any lobbyist.
Unless you have the money to run them locally, and if you do, you could do a lot worse than scams. Ask any lobbyist.
Amazing how we quickly the crypto wars have been forgotten and the lessons not learned.
"But what about criminals?! What about terrorists?! Won't someone please think of the children?!!!"
Every argument against open source AI is moot. Outside of totalitarianism, you cannot control people building algorithms with math and software.
And even that won't work in the long run. People are not going to stop their AI printing presses because the Church of Venture Capital needs regulatory scarcity to command valuations.
Every argument against open source AI is moot. Outside of totalitarianism, you cannot control people building algorithms with math and software.
And even that won't work in the long run. People are not going to stop their AI printing presses because the Church of Venture Capital needs regulatory scarcity to command valuations.
> Outside of totalitarianism
The flaw in your argument is assuming that decision makers don’t want this
The flaw in your argument is assuming that decision makers don’t want this
Ah. But which decisionmakers?
Would it be the ones that tried to backdoor SSL and encryption in the 90s?
Would it be the ones who argued for censoring speech on social platforms?
Do those individuals in both cases share the same political ideology as the Bay Area?
I'll provide a counter thought experiment. What if, a certain politician whose campaign was derailed by people who censored him and kicked him off every tech platform simultaneously... thought it would be very funny to "commoditize his complement"?
Would it be the ones that tried to backdoor SSL and encryption in the 90s?
Would it be the ones who argued for censoring speech on social platforms?
Do those individuals in both cases share the same political ideology as the Bay Area?
I'll provide a counter thought experiment. What if, a certain politician whose campaign was derailed by people who censored him and kicked him off every tech platform simultaneously... thought it would be very funny to "commoditize his complement"?
Begun, the ai wars have.
It's already an established industry with human resources. If you read news about how these scam factories operate in Southeast Asia, their cost to use real humans is even cheaper than running a 2.8T model locally for scams. You don't need genius-level intelligence to run scams. Maybe we tend to overestimate how smart people are.
Nothing, which is already happening with weaker models which are already released.
their reputation, future and business and the fact that people can go somewhere else, just like anywhere else in the open source world. What stops RedHat from putting genius level scams into their operating systems? The fact they're done after they'd pull of their first scam.
Obviously if you're using something anonymously published to the internet you can get scammed the same way you can get scammed on Craigslist, so just don't do that
Obviously if you're using something anonymously published to the internet you can get scammed the same way you can get scammed on Craigslist, so just don't do that
The most compelling argument against Open Source AI: it is analogous to "open sourcing" nuclear weapon technology. Or an how-to guide for making meth/bombs/bioweapons. There is a high risk that we will one day achieve ASI or something close to it. When that day nears, if even one open-source AI user handles their AI in an irresponsible way, we are all FUBARed. In the same way that one irresponsible person with a nuclear bomb can FUBAR the entire world, hence why "open sourcing nuclear weapon tech" is not even a subject of debate.
You don't have to believe that ASI is 100% likely. People like Hinton believe there's a 50% chance of existential risk from AI, but you don't have to go that far either. If there's even a tiny chance of ASI posing a threat to humanity, that's reason enough to lock it down
You don't have to believe that ASI is 100% likely. People like Hinton believe there's a 50% chance of existential risk from AI, but you don't have to go that far either. If there's even a tiny chance of ASI posing a threat to humanity, that's reason enough to lock it down
The difference is that people building atomic weapons knew what they were building. OpenAI/Anthropic have no idea how emergent intelligence works. They're just running in the dark towards the cliff. Leaving it in their hands is the same risk as leaving it to any random guy, because random guy and the labs have the same understanding of what they're building. I'm tired of us treating labs like they're somehow uniquely good at AI. They are not. They simply have enough capital to train. If training was affordable they would have no edge whatsoever.
This argumentation is going to be joked about in the future. Like downloading a car.
Would you open source a nuclear bomb???
Would you open source a nuclear bomb???
I'm in favor of open source AI... however:
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ```
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ```
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.
And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)
And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)
> going on pure trust
The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.
The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.
Trust shifts to the producer of the open weight model, based on the global communicated experience of all models they’ve ever produced.
This doesn't make any sense! Arguing for the sake of arguing.
Why is Windows not open source then, where we can trust Microsoft based on the experience with previous Windows versions?
Why is Windows not open source then, where we can trust Microsoft based on the experience with previous Windows versions?
Exactly. Faced with a choice between open weights that might be poisoned and walled-off weights that I know are poisoned... well, it's an easy choice.
> It isn't trivial to tell if the weights have poisoned:
True, but it is easier if you have the weights than if you don't. I guess this was their point?
True, but it is easier if you have the weights than if you don't. I guess this was their point?
Yes that would be my view. Open access doesn't make it trivial, but remains the most expedient way to remediate these exploits.
You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.
I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.
I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.
[deleted]
From a neutral standpoint, governments usually have more general, comprehensive analytics, as well as greater context and insight into the risks of AI usage, and a possibly better methodology to control access to or use of open-source AI (perhaps by analyzing digital activities and consequences), because they learn from failures. On the other hand, open source creates major opportunities, and it’s hard to accept bans on something that truly benefits users acting for the common good. However, it also benefits unethical or opportunistic users—for whom regulation is indeed necessary.
The 'arguments' are not meant to persuade, they are meant to establish and entrench an overton window (regarding 'AI' in this case).
Tom doesn't take up the single argument that is of primary concern to me,
which is that capable models running in environments where they can be and regularly are abliterated,
and as a society and civilization, we have no good answer to what to do about the destabilizing threats this poses.
I don't have an answer.
And I'm not arguing in favor of some cynical solution which distills down to the state defending oligopolies.
But the threats are real and we have to reason about them soberly, widely, loudly, and quickly.
Most readers here know that defense is always at a disadvantage when it comes to security. That is true to an extent hard to fathom given the innumerable dimensions along which an antisocial actor can apply the force-multiplication of capable models to ill ends. It doesn't have to be FUD-adjacent concerns such as biological terrorism and cybersecurity, though these are real; the opportunities for mischief and misadventure are endless and the opportunities for poisoning the body politic or bringing down basic infrastructure many.
What then is to be done...?
I don't know; but I do know that the argument that almost all use of open models poses no threat is insufficient.
We have not reckoned as a culture with force-multiplication such as this technology brings.
We need to get ahead of the curve, and that will require making novel hard and unwelcome decisions in the short term.
The IMO inevitable outcome otherwise is to rue in leisure, after some global shock, assuming we have opportunity to.
which is that capable models running in environments where they can be and regularly are abliterated,
and as a society and civilization, we have no good answer to what to do about the destabilizing threats this poses.
I don't have an answer.
And I'm not arguing in favor of some cynical solution which distills down to the state defending oligopolies.
But the threats are real and we have to reason about them soberly, widely, loudly, and quickly.
Most readers here know that defense is always at a disadvantage when it comes to security. That is true to an extent hard to fathom given the innumerable dimensions along which an antisocial actor can apply the force-multiplication of capable models to ill ends. It doesn't have to be FUD-adjacent concerns such as biological terrorism and cybersecurity, though these are real; the opportunities for mischief and misadventure are endless and the opportunities for poisoning the body politic or bringing down basic infrastructure many.
What then is to be done...?
I don't know; but I do know that the argument that almost all use of open models poses no threat is insufficient.
We have not reckoned as a culture with force-multiplication such as this technology brings.
We need to get ahead of the curve, and that will require making novel hard and unwelcome decisions in the short term.
The IMO inevitable outcome otherwise is to rue in leisure, after some global shock, assuming we have opportunity to.
The way Anthropic is playing the "AI is scary and dangerous and only we can be trusted with it" game is clearly aimed at regulatory capture, which would only be good for Anthropic. It's worrying how cozy some of the tech leaders have been with the Trump administration, which is also clearly aimed at regulatory capture that serves the interests of the biggest tech companies and no one else.
The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
Just because encryption bans are badly conceived or implemented doesn't mean they are wrong.
Encryption has military value.
The point is that the good outweighs the bad. Making the government seem like buffoons for attempting to prevent military technology like dual use crypto is not in our best interest.
The same government upheld the right to free speech, so this balancing act is widely observed.
The government is not evil, and yes they make mistakes, but they also not infrequently protect us.
And I, for one, like being protected.
Encryption has military value.
The point is that the good outweighs the bad. Making the government seem like buffoons for attempting to prevent military technology like dual use crypto is not in our best interest.
The same government upheld the right to free speech, so this balancing act is widely observed.
The government is not evil, and yes they make mistakes, but they also not infrequently protect us.
And I, for one, like being protected.
Regarding encryption. Interestingly, US government also insisted on DES and AES being open standards, because using bad encryption did more harm than good.
> OpenAI's Dean Ball:
> One probable outcome of an open-weight-model-dominant world is full AI communism...
This quote proves me that there are no rational arguments here. It’s just Cold War era mentality. This is the first time US is really challenged in technological dominance since the Soviets. Sadly this seems like the only playbook Americans still use. This is their modus operandi. This is how they handle competition and they haven’t updated their playbook since 80s.
There is nothing necessarily malicious about open weight models. Being “Chinese” is the real thing which makes them “malicious”.
It seems public still buys this rhetoric. And leaders and policy makers still use it because they don’t know any better. But I have to say outside US it makes no sense in 2026’s world.
> One probable outcome of an open-weight-model-dominant world is full AI communism...
This quote proves me that there are no rational arguments here. It’s just Cold War era mentality. This is the first time US is really challenged in technological dominance since the Soviets. Sadly this seems like the only playbook Americans still use. This is their modus operandi. This is how they handle competition and they haven’t updated their playbook since 80s.
There is nothing necessarily malicious about open weight models. Being “Chinese” is the real thing which makes them “malicious”.
It seems public still buys this rhetoric. And leaders and policy makers still use it because they don’t know any better. But I have to say outside US it makes no sense in 2026’s world.
I’m still kind of shocked that Dean Ball can tweet such incendiary stuff about OpenAI policy. Like presumably OpenAI would prefer it if their staff don’t pick fights with Trump administration officials.
designs on the time scale I guess. and openai might feel secure in their favor with the admin?
> One probable outcome of an open-weight-model-dominant world is full AI communism...
oh no, not communism.
oh no, not communism.
They don't have to convince developers, they just have to convince the general population. I've had several discussions about open source with people who don't care about coding and it's hard to untangle the misinformation they receive from the media. They get talking points from authorities that they don't understand so they will always double down on it. For example, rhetoric relating open source to communism is humiliating to discuss. For proprietary evangelists, the cruelty is the point.
"For example, rhetoric relating open source to communism is humiliating to discuss"
All Open Source isn't communism. The GPL license is a form of digital communism. It forces you to open source any additions made to open source code as a way to make things 'equal'.
I'm glad it and Stallman are mostly in the dust heap of history and better licenses, like the MIT, BSD, and Apache have gotten popular.
The irony is that the only open source projects surviving long term are funded by very large corporations.
If you don't care about creating code, none of this should really matter.
All Open Source isn't communism. The GPL license is a form of digital communism. It forces you to open source any additions made to open source code as a way to make things 'equal'.
I'm glad it and Stallman are mostly in the dust heap of history and better licenses, like the MIT, BSD, and Apache have gotten popular.
The irony is that the only open source projects surviving long term are funded by very large corporations.
If you don't care about creating code, none of this should really matter.
> The irony is that the only open source projects surviving long term are funded by very large corporations.
The real irony is that what is probably the most important open source project in history has a GPL license.
Dust heap of history my ass.
The real irony is that what is probably the most important open source project in history has a GPL license.
Dust heap of history my ass.
Reciprocal sharing is not "digital communism". If it was, then the mere existence of the public domain and the fact that copyrights expire is "delayed communism".
The GPL was absolutely necessary in getting corporate America to play ball with FOSS projects instead of merely extracting value from them. It's also the only reason why embedded vendors even release anything at all. The licenses on Linux and busybox require it. The fact that permissive licenses are more common for certain kinds of FOSS projects does not mean the GPL is dead - far from it.
Stallman is more or less irrelevant, but that's because Stallman let GNU become irrelevant. Without GNU, he's a washed-up has-been from the MIT AI Lab with a startlingly low Epstein number[0], a chain of disturbing accusations, and a foundation that acts like his personal sockpuppet. The reason why companies still play ball with Linux and it's GPL license is because Linux continues to deliver a Free and free kernel worth using at the low low price of "please send back your source code changes". Which, again, is not "digital communism", it's just a good bargain.
[0] Erdos number but for pedophiles
The GPL was absolutely necessary in getting corporate America to play ball with FOSS projects instead of merely extracting value from them. It's also the only reason why embedded vendors even release anything at all. The licenses on Linux and busybox require it. The fact that permissive licenses are more common for certain kinds of FOSS projects does not mean the GPL is dead - far from it.
Stallman is more or less irrelevant, but that's because Stallman let GNU become irrelevant. Without GNU, he's a washed-up has-been from the MIT AI Lab with a startlingly low Epstein number[0], a chain of disturbing accusations, and a foundation that acts like his personal sockpuppet. The reason why companies still play ball with Linux and it's GPL license is because Linux continues to deliver a Free and free kernel worth using at the low low price of "please send back your source code changes". Which, again, is not "digital communism", it's just a good bargain.
[0] Erdos number but for pedophiles
I would be more inclined to agree if American companies were not in a competition to be the biggest flameout and lose the most money. How is China able to afford to train models that US companies need to burn billions of dollars for?
In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democratic socialist" if there is no capitalist reform.
In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democratic socialist" if there is no capitalist reform.
It's awfully rich and sad to see "Open" AI argue against open weight model releases. I guess the employees at "Open" AI have the nerve to believe that releasing GPT 3's weights really would have been too dangerous, so they simply had to pull it. Still, even though it wasn't believable back then, either.
Now, of course, according to "Open" AI, the only two outcomes is that we have an oligopoly of regulatory-capture-protected U.S. AI companies that dispense compute in the form of tokens that you have to pay for, or "AI communism". This is, according to that OpenAI employee, after all what everyone including open weight advocates believe the outcome must be.
Notably they seem to be excluding the possibility that maybe personal computers could still become exponentially more powerful and resourceful as they have done many, many, many times in the past. Are you absolutely sure there's no way that a computer in 10 years won't just be able to run a Kimi K3 quantization? I'll admit that scaling limits will probably continue to slow down growth in computers, but idk, do we really believe we're completely out of breakthroughs and major incremental improvements already? What happens to all of those data center parts when they become "obsolete" because they're not dense/efficient enough anymore. Do we burn them to make sure they can't be used by someone else?
Or perhaps they believe that the "baseline" models of the future will just be 100+ T parameters and again still require datacenter equipment to run. I hate to be the "640k ought to be enough for anybody" guy but I just don't find this terribly compelling. For larger parameter models, training costs scale quadratically at least, whereas inference costs scale linearly. To me it seems with better computers the consumer and enthusiast tier devices will always be shifting towards being able to do inference on things that are trained using increasingly more compute power over time.
I hate to seem like too much of an optimist in what seems like an awful situation, but I just can't see how this will go down well for OpenAI and Anthropic. China's strategy is working because our strategy is shit and makes no sense, and they don't seem to be backing away from it. Anthropic came pretty close to saying the quiet part out loud with their calls for a framework to "pause" AI development globally. If you read between the lines, it's hard to pretend you don't know why they might want that. (It has to do with what happens if you want to stop losing money training ever larger models, without falling behind in a competitive market.)
The way I see it, China has put us in a Chinese finger trap scenario. Harder we tug, worse it's gonna get. And to be honest, knowing how we are, we're probably going to tug. Will that wind up lasting to another administration? We'll find out, but it's certainly possible.
Now, of course, according to "Open" AI, the only two outcomes is that we have an oligopoly of regulatory-capture-protected U.S. AI companies that dispense compute in the form of tokens that you have to pay for, or "AI communism". This is, according to that OpenAI employee, after all what everyone including open weight advocates believe the outcome must be.
Notably they seem to be excluding the possibility that maybe personal computers could still become exponentially more powerful and resourceful as they have done many, many, many times in the past. Are you absolutely sure there's no way that a computer in 10 years won't just be able to run a Kimi K3 quantization? I'll admit that scaling limits will probably continue to slow down growth in computers, but idk, do we really believe we're completely out of breakthroughs and major incremental improvements already? What happens to all of those data center parts when they become "obsolete" because they're not dense/efficient enough anymore. Do we burn them to make sure they can't be used by someone else?
Or perhaps they believe that the "baseline" models of the future will just be 100+ T parameters and again still require datacenter equipment to run. I hate to be the "640k ought to be enough for anybody" guy but I just don't find this terribly compelling. For larger parameter models, training costs scale quadratically at least, whereas inference costs scale linearly. To me it seems with better computers the consumer and enthusiast tier devices will always be shifting towards being able to do inference on things that are trained using increasingly more compute power over time.
I hate to seem like too much of an optimist in what seems like an awful situation, but I just can't see how this will go down well for OpenAI and Anthropic. China's strategy is working because our strategy is shit and makes no sense, and they don't seem to be backing away from it. Anthropic came pretty close to saying the quiet part out loud with their calls for a framework to "pause" AI development globally. If you read between the lines, it's hard to pretend you don't know why they might want that. (It has to do with what happens if you want to stop losing money training ever larger models, without falling behind in a competitive market.)
The way I see it, China has put us in a Chinese finger trap scenario. Harder we tug, worse it's gonna get. And to be honest, knowing how we are, we're probably going to tug. Will that wind up lasting to another administration? We'll find out, but it's certainly possible.
You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.
My level of trust in Anthropic/Google/OAI/Microsoft/Meta not to do bad things is about zero, so why should I trust them with this?
These companies are using anything at hand to trick people out of their money … be very careful…
“We clearly can’t “trust” these companies not incentivized to randomly kill lots of people, so we might as well give everyone the ability to randomly kill lots of people”.
So? The primary thing stopping people from doing really really bad things is, and always has been, most people not wanting to do bad things.
It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it.
The reason you haven't rammed a pen into your coworker's neck is the same reason why most possible really bad things don't happen: People don't want to do them.
It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it.
The reason you haven't rammed a pen into your coworker's neck is the same reason why most possible really bad things don't happen: People don't want to do them.
It’s interesting that most of the examples people making this argument give are high consequence, high difficulty (could you actually murder your coworker with a pen? I’m skeptical.) low kill count methodologies of violence.
We’re specifically talking about low difficulty, highly effective, high kill count methodologies of violence.
And yes, being able to kill a lot of people trivially, even with huge consequences is something people do. Most countries don’t let you run around with an armory as a result of that.
We’re specifically talking about low difficulty, highly effective, high kill count methodologies of violence.
And yes, being able to kill a lot of people trivially, even with huge consequences is something people do. Most countries don’t let you run around with an armory as a result of that.
I live in one of the exceptions to that, and still have zero concern that someone's going to shoot up my workplace. Again, civilization works because the vast majority of people are utterly repulsed by the idea of harming another person.
Most of the hypotheticals that people come up with are things that could pretty trivially done by a mad scientist today. But that doesn't happen, because this isn't a comic book, there is nobody hell-bent on turning everyone into dinosaurs.
Most of the hypotheticals that people come up with are things that could pretty trivially done by a mad scientist today. But that doesn't happen, because this isn't a comic book, there is nobody hell-bent on turning everyone into dinosaurs.
> Most countries don’t let you run around with an armory as a result of that.
My country allows it!
My country allows it!
I already know how to kill thousands of people. It's not conceptually difficult: more than a day's effort, sure, but what stops people from doing this kind of thing is not the lack of knowing how. https://xkcd.com/1958/
No, you don't. What stops people is actually access to knowledge. There are killers of varying levels of efficacy; making killing easier means more people die. It's 1-1.
How many people have shot up electrical substations, a fairly trivial way to causing widespread harm that was popularized in the news just a few years ago.
But it's only happened a couple times. Because most people just can't be assed to harm a hundred thousand others.
But it's only happened a couple times. Because most people just can't be assed to harm a hundred thousand others.
Most people don’t want to hurt lots of other people in an untargeted way, yes.
But there are a lot of people, and, apparently a few of them shot up some electrical substations?
If you hold constant the (very small) fraction of people who want to cause mass harm, and increase the average ease with which a randomly selected person could cause mass harm , you would expect the amount of deliberate mass harm to increase, even if the fraction of people who would want to do that is like, 0.000001 of people?
But there are a lot of people, and, apparently a few of them shot up some electrical substations?
If you hold constant the (very small) fraction of people who want to cause mass harm, and increase the average ease with which a randomly selected person could cause mass harm , you would expect the amount of deliberate mass harm to increase, even if the fraction of people who would want to do that is like, 0.000001 of people?
The kind of person who wants to cause mass harm is the kind of person who's thought about it for more than 15 minutes. If you think about it for more than 15 minutes, I'm sure you can come up with plenty of ways. (Even 5 minutes is probably long enough to produce 3 or 4 workable suggestions.)
If AI causes people to kill who otherwise wouldn't have, I expect it to be by causing and exacerbating mental health problems, not by providing knowledge. (See e.g. https://www.bbc.co.uk/news/technology-67012224, where a roleplay escalated into a concrete plan to kill Queen Elizabeth II over the course of two weeks. The plan wasn't, however, a very good one.)
If AI causes people to kill who otherwise wouldn't have, I expect it to be by causing and exacerbating mental health problems, not by providing knowledge. (See e.g. https://www.bbc.co.uk/news/technology-67012224, where a roleplay escalated into a concrete plan to kill Queen Elizabeth II over the course of two weeks. The plan wasn't, however, a very good one.)
> What stops people is actually access to knowledge.
Very untrue.
I'm a combat vet. I spent years fighting an insurgency and therefore pretty good at that very task. I have the knowledge, so what stops me?
Very untrue.
I'm a combat vet. I spent years fighting an insurgency and therefore pretty good at that very task. I have the knowledge, so what stops me?
FYI combat vets are a counter example. They frequently kill a lot of people effectively. See: the DC sniper.
There are around 16 million living veterans.
Between 4.5-6 million living US combat veterans.
5-8% of all mass casualty perpetrators are a combat veteran, between 92-95% of all mass casualty perpetrators are not combat veterans
By far, there are more veteran suicides than there are mass casualties caused by combat veterans. Veterans are more likely to kill themselves than to kill you.
DC sniper was 24 YEARS ago, prior to the invasion of Iraq during the GWOT. Not only that, John Allen Muhammad didn't even see combat, his job was to dismantle old rockets! He was a mechanic and truck driver!
The point of discussion is: "What stops people is actually access to knowledge."
Statistics show this to be clearly false: Knowledge is not required to be effective in causing a mass casualty event, in fact, lack of formal training is a higher predictor!
Between 4.5-6 million living US combat veterans.
5-8% of all mass casualty perpetrators are a combat veteran, between 92-95% of all mass casualty perpetrators are not combat veterans
By far, there are more veteran suicides than there are mass casualties caused by combat veterans. Veterans are more likely to kill themselves than to kill you.
DC sniper was 24 YEARS ago, prior to the invasion of Iraq during the GWOT. Not only that, John Allen Muhammad didn't even see combat, his job was to dismantle old rockets! He was a mechanic and truck driver!
The point of discussion is: "What stops people is actually access to knowledge."
Statistics show this to be clearly false: Knowledge is not required to be effective in causing a mass casualty event, in fact, lack of formal training is a higher predictor!
Photoshop source code + OSI license = open source
Photoshop binary = open weight
Photoshop SAAS web app = closed model like GPT, Opus/Fable etc.
There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API.
If you want actual open source then you would need to look at like OLMo 3
https://allenai.org/