That’s usually not true due to caching. It may be true if you leave a large gap in between, but if you send “make it red” right after, then it’s purely incremental
Exactly. This number is so so much bigger than 10^100000 or however many grains of sand would fit, that dividing by that amount doesn’t really change it, certainly not enough to bring it down closer to 9,999,999sub10
The two other changes they mention have been widely adopted, and are included in at least some of the models they benchmark against. It seems they list them for completeness as changes to the original transformer architecture.
This is just an artefact of tokenisation though. The model simply isn’t ever shown the letters that make up words, unless they are spelled out explicitly. It sees tokens representing groups of words. This is a little like saying a human isn’t intelligent because they couldn’t answer your question that you asked in an ultrasonic wavelength. If you’d like to learn more this video is a great resource: https://youtu.be/zduSFxRajkE?si=LvpXbeSyJRFBJFuj
I used to use K professionally inside a hedge fund a few years back. Aside from the terrible user experience (if your code isn’t correct you will often just get ‘error’ or ‘not implemented’ with no further detail), if the performance really was as stellar as claimed, then there wouldn’t need to be a no benchmark clause in the license.
It can be fast, if your data is in the right formats, but not crazy fast. And easy to beat if you can run your code on the GPU.
For anyone wondering what this does, it looks like it produces optimal configurations for belt balancers given a specified number of input and output belts. Belt balancers evenly distribute items between belts: https://wiki.factorio.com/Balancer_mechanics
Nice paper. I particularly like how they talk through the ideas they tried that didn’t work, and the process they used to land on the final results. A lot of ML papers present the finished result as if it appeared from nowhere without trial and error, perhaps with some ablations in the appendix and I wish more papers followed this one in talking about the dead ends along the way.
That’s only true for linearly ordered structures, but isn’t true for partially ordered ones.
For example, set inclusion. Two different sets can be neither greater than not smaller than each other. Sets ordered by inclusion form a partially ordered lattice.
From reading this book you’d have a very good grasp of the underlying theory, much more than many ML engineers. But you’d be missing out on the practical lessons, all the little tips and intuitions you need to be able to get systems working in practice. I think this just takes time and it’s as much an art as it is a science.
The other angle on ‘Corporate AI’ is when we’ll start to see product placement and adverts inside generated content. Create an image of coffee, and you’ll find Starbucks logos everywhere. Ask an LLM about a topic and see it work in an advert about a particular brand of beer. I’m sure people are working on this already, but I really hope it never happens.
Of course they’re different. But so what? That’s not exactly proof of anything, unless you’re suggestion biological neurons are the only configuration in the universe capable of thought? Maybe that’s true, but it seems unlikely to me.
The pressure of natural selection can lead to the phenomenon of consciousness. Why not the process of training llms? Perhaps developing the machine equivalent of consciousness helps that particular configuration of weights survive the otherwise destructive process of gradient descent.
This paper, and most other places i’ve seen it argued that language models can’t possibly be conscious, sentient, thinking etc, rely heavily on the idea that llms are ‘just’ doing statistical prediction of tokens.
I personally find this utterly unconvincing. For a start, I’m not entirely sure that’s not what I’m doing in typing out this message. My brain is ‘just’ chemistry, so clearly can’t have beliefs or be conscious, right?
But more relevant is the fact that llms like ChatGPT are only pre-trained on pure statistical generation, followed by further tuning through reinforcement learning. So ChatGPT is no longer simply doing pure statistical modelling, though of course the interface of calculating logits for the next token remains the same.
note: i’m not saying i think llms are conscious. I don’t think the question even makes much sense. I am saying all the arguments that i’ve seen for why they aren’t have been very unsatisfying.