It showed me that a model trained only on movie subtitles data exhibited some (very primitive) reasoning. I have been working on Deep Learning and later LLMs ever since.
Europe is quite conservative, in the sense that they would not invest billions into an unproven venture. It makes sense that it would excel at an industry that requires putting safety above everything.
It's actually true on many levels, if you think about is needed for generating syntactically and grammatically correct sentences, coherent text and working code.
You can also look at the price of opensource models on openrouter, which are a fraction of the cost of closed source models. This is a market that is heavily commoditized, so I would expect it reflect the true cost with a small margin.
Presumably the model is trained in post-training to produce a response to a prompt, but not to reproduce the prompt itself. So if you prompt it with an empty prompt it's going to be out of distribution.
The study seemed not very convincing to me, at least the way it was described in the article. To summarize: they asked crowdworkers to write a law who used legalese, but not when writing news stories about it or when explaining the law. From that the researchers concluded that people use legalese to convey authority.
But what if people just imitated the writing style of existing laws, but not with the intention to make it authoritative but because that is what they understood their task to be?
I have an upcoming trip to Europe, which I am quite excited about. I wanted to set up a Tailscale exit node to ensure that critical apps I depend on, such as banking portals continue working from outside the country.
I've never had an issue accessing banking portals from Europe.
No this is not correct. Arguably OpenAI invented LLMs with GPT3 and the preceding scaling laws paper. I worked on LAMDA, it came after GPT4 and was not as capable. Google did invent the transformer, but all the authors of the paper have left since.
Yes, I think that is a reasonable way to think about it, in my opinion. However, with the language modeling objective it predicts the next token and because of the residual connections each intermediate layer is in the same space. So, maybe it would be more accurate to say that it is an increasingly accurate representation of the next token.
Attention takes in all tokens in the sequence and outputs a new representation of the current token in context. Each layer of the transformer adds more context to the token.
I haven't read this explanation in detail and although they have some nice animations, I wouldn't go to FT to explain machine learning concepts. Here are two well known explanations that might be better:
It showed me that a model trained only on movie subtitles data exhibited some (very primitive) reasoning. I have been working on Deep Learning and later LLMs ever since.