Thanks for writing this so clearly... I hear wrong/misguided arguments like we see hear every day from friends, colleagues, "experts in the media" etc.
It's strange because just a moment of thinking will show that such ideas are wrong or paint a clearly incomplete picture. And there's plenty of analogies to the dangers of such reductionism. It should be obviously wrong to anyone who has at least tried ChatGPT.
My only explanation is that a denial mechanism must be at play. It simply feels more comfortable to diminish LLM capabilities and/or feel that you understand them from reading a Medium article on transformer-network, than to consider the consequences in terms of the inner black-box nature.
You write: "it's just that at every step the alligator is lurking in the logits because it directly derives from the prompt" - but isn't that the whole point: at the moment the model writes "an", it isn't just spitting out a random article (or a 50/50 distribution of articles or other words for that matter); rather, "an" gets a high probability because the model internally knows that "alligator" is the correct thing after that. While it can only emit one token in this step, it will emit "an" to make it consistent with its alligator knowledge "lurking". And btw while not even directly relevant, the word alligator isn't in the prompt. Sure, it derives from the prompt but so does every an LLM generates, and same for any other AI mechanism for generating answers.
I don't understand how OpenAI claims it would have happened. The weights are closed and as far as I read they are not complaining Deepseek hacked them and obtained the weight. So all they could do was to query OpenAI and generate test data. But how much did they query really - I would suppose it would require a huge amount done via an external, paid-for API? Is there any proof of this besides OpenAI saying it? Even if we suppose it is true, I suppose this must have happened via the API so they paid per token etc. So they paid for each and every token of training data. As I understand, the requester owns the copyright on what is generated by OpenAI's models and is free to do what they want.
You have to give it a chance. He is first building up an argument about why consciousness cannot depend just on the physical substrate itself but rather on the "interpretation" of this. It is very important to understand this part/argument. What follows is something that resonates with you namely how our consciousness is now 'tuned' to the current physical laws.
I'm wondering how deep the hack is... it seems sending a frame is just setting some registers and waiting for an interrupt. This suggests (though I'm not an expert!) that they are talking to another layer of firmware that does the actual stuff? Reminds me a bit of the Raspberry pico board which has the main RP2040 SoC but where the WiFi is a separate WiFi/BT module (CYW43xx) with its own Arm cores. Not even the external register interface to the WiFi module is documented publicly, but there is an open source driver (https://github.com/georgerobotics/cyw43-driver/tree/cf924bb0...) so one can infer the specification. However, this driver yet again talks to software running on Arm codes inside the module, the code for which is supplied as big firmware binary blobs by the manufacturer (the blobs are actually in the linked repo, defined inside header files in the firmware directory). I'm wondering how this ESP32 hack corresponds to this?
It's strange because just a moment of thinking will show that such ideas are wrong or paint a clearly incomplete picture. And there's plenty of analogies to the dangers of such reductionism. It should be obviously wrong to anyone who has at least tried ChatGPT.
My only explanation is that a denial mechanism must be at play. It simply feels more comfortable to diminish LLM capabilities and/or feel that you understand them from reading a Medium article on transformer-network, than to consider the consequences in terms of the inner black-box nature.