I have been a pretty consistent user of AI since 2022 (Instruct-GPT), so I don't have a bad opinion about the topic. However, I think the real problem now has become pretty obvious. We are hitting a reality wall, where we simply don't have enough ressources to feed the AI industry. We don't generate enough electrical power nor enough GPU or TPU. For the first time in computer science, the real issue here is the finitude of the physical world. Unless, we start digging asteroids, we are already facing a shortage of raw material and industrial output. In my opinion, the only way to go is small models running on regular hardwares.
LispE is a programming language that has been ported to WASM. It's lightweight (about 3.3MB) and provides many features natively, with no external dependencies. You can now try it directly in your browser.
This article nails it. The claim 10x is in my opinion one of these tactics used by large corporations to force engineers into submission. The idea that you could be replaced with an AI is frightening enough to keep people in check, when negotiating your salary. AI is a wonderful tool that I use everyday, and I have been able to implement stuff that I would have considered too cumbersome to even start working on. But, it doesn't make you a 10x more efficient engineer. It gives you an edge when you start a new project, which is already a lot. But don't expect your whole project of 100,000 lines to be handled by the machine. It won't happen any time soon.
I love how so many people are eager to criticize LLM code, when in fact, according to my experience it is pretty superior to anything I have seen produced by human programmers, most of the time. It is documented, the code is explained at each step of its creation, and it is pretty readable when you dig into it. I have 30 years of experience in coding, and I have been playing with these LLM for 3 years. Yeah!!! Of course, sometimes they produce very bad code. But in average, the code they produce is largely on par with my fellow humans. And since, they produce the whole explanation of it, it takes a couple of minutes to understand it. And if you don't understand the main points of the code, the LLM will tell you all about it. When did you have a colleague that was eager to explain his/her code to you??? When did you have a colleague that did produce a code you could understand in a few minute??? I really think these tools are quite useful, no need to wrap yourself into the mantel of expertise and look down on these LLM, because sometimes they will produce a code you don't like.
The answer is very easy. Latinate words (mostly of French origin for the matter), first because of the number of people speaking romance languages (French, Italian, Spanish, Romanian and Portuguese) is the largest in Europe, second, because most European languages have also borrowed a lot from Latin, Greek and French during the last centuries, which means that latinate words are usually the subset which is shared across most languages. Furthermore, English and other Germanic languages have started evolving quite early one from the others (around the 6 century) and cognates might be quite difficult to recognize: through/durch, for instance.
Funny as they always seem to forget the hardware side of search engines. Google was incredibly fast compared to its competitors because they were among the first to store their whole index in RAM rather than on hard drives. They were among the first to install huge data center with computer blades that could be changed in an instant in case of failure. As an early user, I was on board as early as 99, I was amazed by the response speed of Google and its bare style quite dépouillé.
Some people still think that LLM are just word predictors. Technically, it is not. First, transformer architectures don't process words, they process semantic representations stored as vectors or embeddings in a continuous space.
What a lot of people don't understand is that in an LLM, we go from discrete values (the tokens) to continuous values (embeddings) that the transformer takes as input. A transformer is a polynomial function that will project into the latent embedding space. It doesn't generate word per se, but a vector that is then compared against the latent embedding space to find the closest matches. The decoding part is usually not deterministic.
This huge polynomial function is the reason we can't understand what is going on in a transformer. It doesn't mimick human speech, it builds a huge representation of the world, which it uses to respond to a query. This is not a conceptual graph, it is not a mere semantic representation. It is a distillation of all the data it ingested. And each model is unique as the process itself is split over hundreds of gpu, with no control over which GPU is going to churn out which part of the dataset and in which order.
Gradient descent is to AI, what a loop is to understand programming. However, understanding a loop doesn't mean you can program a full video game from scratch. Organizing hundreds of layers in an efficient way is pretty complex, even if the work today has been simplified thanks to PyTorch or Tensorflow, it remains pretty complicated. You have to understand how to organize your data, how to size your batches, how to make your code resilient enough to survive GPU cards crashing. Train a model over hundreds of GPU is really really complicated. New algorithms are proposed all the time, because we have no idea how to handle these interconnected layers in an efficient way, but with cumbersome heuristics. However, salary inflation is never a good thing, because it will create a gap between decent engineers and other people pronounced geniuses. The AI teams will suffer from these decisions... Badly. It will be like these samouraïs who would kill peasants after a battle to increase their head count, because this was how people were rewarded after a battle. Some of these people, in order to justify their salaries, will feel pressure to poach other people's ideas...
The problem of measuring intelligence is that most of these techniques reveal more about what we think intelligence is in a specific cultural context. Which means that individuals who rank higher in these tests are those who conform the most to a society expectancies. High level of literacy, proper nutrition during their childhood, proper education, help foster better IQ. Which of course leads to a better understanding of how your own society works, hence to a better evaluation of probabilities. If you have grown up in a family with money, you have a much better understanding of how to benefit from your investments.
What I do is to build a substring out of the initial strings that is a multiple of 2. If the string I try to search is 9 characters long, then I extract an 8 characters substrings that I transform into an integer over an integer:
Here is the example for 4 characters:
//We look for the presence of four characters in a row
```
int32_t cc = search[3];
for (shift = 2; shift >= 0; shift--) {
cc <<= 8;
cc |= search[shift];
}
__m256i firstchar = _mm256_set1_epi16(cc);
```
In this case, I will look for a 4 bytes integers over my sequence:
```
current_bytes = _mm256_cmpeq_epi16(firstchar, current_bytes);
q <<= 1;
q |= _mm256_movemask_epi8(current_bytes);
````
I'm looking for blocks of 4 characters at a time in my string.
Seriously, most fine-tuning now is done with LoRa adapters. They are much faster and more reliable. In my lab, I don't know anybody who is trying to do any kind of thorough fine-tuning...
As I said many times, I have been in the game for 30 years. I started doing AI with rules as early as the beginning of the 90 and I never...never expected to see anything like LLMs in my lifetime.
When I read Marcus once again saying that: yes this time LLM have reached their limit, which he has been saying for 2 years in a row, I'm really feeling tired of his tune. The idea that LLM are a dead end, a failing technology is pretty weird.
Compared to what???
I use LLM everyday in my work, to write summaries, to make translations, to generate some code or to get explanations about a given code... And I even use them as research sparring partners to see how I could improve my work...
Gary Marcus has been involved in the domain for 30 years as well... Where is his technology that would match or surpass the LLM???