> They want you to rent their compute instead of running it on your own hardware
I don't think this is the main thing. I think the real risk open weights AI poses to them is that it allows us to get inference from providers that only care about making profits on inference. What OpenAI/Anthropic want is to be able to jack up inference prices once everyone is dependent. Open weights keeps them honest.
What's interesting is that the way in which they're childish is actually extremely human. In fact, one of the ways that you're often taught to draw more realistically is to stop thinking of the concepts as icons you're drawing the outlines of, and instead sort of blur your eyes and see things as they are: hues and values. In other words, become a camera or a printer that has no idea what it's capturing or printing other than a grid of values. That is how you achieve realism.
The fact that it has clearly iconified these concepts in its mind and is tracing the outlines of the things it thinks/expects to go where is very human.
It's funny that they triggered the infamous "Code Red" moment in OpenAI when the 3-3.1 models came out. I switched to using them for a lot of single-shot LLM calls because they were fast and cheap. Their only area that they were lacking in was agentic/tool calling.
Needless to say 3.5 was a disappointment. Curious to see 3.6.
If you want an LLM to have knowledge about something, the knowledge has to either exist in its weights or be provided to it in its context. Because context is expensive and limited, and models tend to get dumber the more their context is filled, there is usually more that you'd need to put into context than can reasonably fit in it in order for the model to answer questions about your data. So your options are basically
1.) stuff it into context
2.) figure out a way to determine what to put into context based off of what is being asked of the model (RAG)
3.) change the weights of the model to have knowledge of your data baked into it (fine tuning)
This is like a single prompt AI. The term labels are illegible against the white background. The connections don't make any sense. "Training vs Inference" connects to Large Language Models and GPU. It doesn't connect to RLHF. Machine Learning connects to fewer things than Large Language Models. The whole thing is insane.
MoE is literally exactly what they're describing. The classifier being described is baked into the model and is the thing that makes it MoE.
Imagine I had [a model that was good at math], [model that was good at code], [model that was good at writing], [model that was good at general knowledge]. If I then had [a model that was good at determining whether the user query would be best served by one of those models and sent it to it, leaving the rest of the models inactive], that is the platonic version of what MoE is. In practice, it works a bit differently. It instead basically restricts the number of pathways that can be utilized in solving problems during training, which allows for "expert neuron groupings" to form and "classifier layers" to form earlier on in the structure, but the effect is the same (better, even, since it allows some overlap between structures of experts). It also allows "routing to an expert" to happen token-by-token rather than at the prompt level.
I didn't know people could just have GPT running on their own hardware. How does one...do that? Do you have a special relationship with OpenAI and they lock down your servers or something?
I have built a few voice based integrations into my applications that use these live agents (gpt and gemini), but they are always too expensive to be viable. I have to end up hacking up context and turning on and off in ways that are very fragile. It'll end up being $2-5 for about the 30ish minute sessions I typically end up with, and it throws the price of the product I'm making completely out of whack.
> This implies that corners were cut. They were not. They went through the full regulatory procedures.
That's absolutely not true. The standard for new vaccines, iirc, required a period of something on the order of 7 years. Time, in this case, is not a function of procedure that can be expedited in an emergency, but is actually an important element in and of itself. Many issues do not manifest immediately and actually need follow up over time.
The crazy thing about these vaccines was that both mRNA vaccines and the viral vector vaccines were completely new platforms, never deployed at scale. They work entirely differently than all other vaccines. Up until this point, vaccines all delivered the antigen in one of 3 ways: you get a weakened virus, you get a dead virus, or you get the antigen itself (subunit protein like Novavax). Both the mRNA (Pfizer & Moderna) and the viral vector (J&J) vaccines worked by getting either mRNA or DNA (viral vector) into your cells, and then having your own cells produce and express the antigen themselves. Basically the difference between server generated code or shipping the JS for you to run the SPA on your own client.
One of the crazy things about this was that it wasn't obvious what the implications would be of having our own cells expressing the antigens (and thus flagging themselves for destruction by our immune system). This was particularly concerning because the cells that were shown to be doing this, despite the complete lie that kept being repeated of the vaccine staying localized at the injection site, were found all over the body. In the case of the viral vector vaccines, at least they were being delivered by a vessel (living adenovirus) that our bodies have had billions of years of evolution to determine where they might end up. In the case of the mRNA vaccines, though, the vessel was a lipid nanoparticle with an exceptional ability to deliver payloads basically anywhere in the body. Note: the attention these lipid nanoparticles had received prior to their use in mRNA was their ability to deliver payloads to places that are notoriously difficult to reach, notably their ability to cross the brain blood barrier. So you have delivery mechanisms delivering a payload that makes our cells into antigen factories, shown to be producing them all over the body, and targeting themselves for destruction by the immune system/causing an increased immune response in these areas.
And then, for the icing on the cake, there was mounting evidence that the antigen itself was actually likely destructive/problematic.
I could go off forever on this topic. The amount of obfuscation and gaslighting was insurmountable for anybody that was even remotely interested in figuring out what was happening. From a personal perspective, my trust in many institutions was permanently shaken.
I don't think this is the main thing. I think the real risk open weights AI poses to them is that it allows us to get inference from providers that only care about making profits on inference. What OpenAI/Anthropic want is to be able to jack up inference prices once everyone is dependent. Open weights keeps them honest.