What can LLMs never do?(strangeloopcanon.com)
strangeloopcanon.com
What can LLMs never do?
https://www.strangeloopcanon.com/p/what-can-llms-never-do
374 comments
> If there exist classes of problems that someone in an elementary school can easily solve but a trillion-token billion-dollar sophisticated model cannot solve, what does that tell us about the nature of our cognition?
I think what it tells us is that our cognition is capable of more than just language modeling. With LLMs we are discovering (amazing) capabilities and the limits of language models. While language models can do incredible things with language that humans can't, they still can't do something simple like sudoku. But there are neural networks, CNNs and RNNs that can solve sudoku better than humans can. I think that the thing to learn here is that some problems are in the domain of language models, and some problems are a better fit for other forms of cognition. The human brain is amazing in that it combines several forms of cognition in an integrated way.
One thing that I think LLMs have the capability to do is to integrate several types of systems and to choose the right one to solve a problem. Teach an LLM how to interface with a CNN that solves sudoku problems, and then ask it a sudoku problem.
It seems to me that if we want to create an AGI, we need to learn how to integrate several different types of models, and teach them how to distribute the tasks we give them to the correct models.
I think what it tells us is that our cognition is capable of more than just language modeling. With LLMs we are discovering (amazing) capabilities and the limits of language models. While language models can do incredible things with language that humans can't, they still can't do something simple like sudoku. But there are neural networks, CNNs and RNNs that can solve sudoku better than humans can. I think that the thing to learn here is that some problems are in the domain of language models, and some problems are a better fit for other forms of cognition. The human brain is amazing in that it combines several forms of cognition in an integrated way.
One thing that I think LLMs have the capability to do is to integrate several types of systems and to choose the right one to solve a problem. Teach an LLM how to interface with a CNN that solves sudoku problems, and then ask it a sudoku problem.
It seems to me that if we want to create an AGI, we need to learn how to integrate several different types of models, and teach them how to distribute the tasks we give them to the correct models.
If we're trying to quantify what they can NEVER do, I think we'd have to resort to some theoretical results rather than a list empirical evidence of what they can't do now.
The terminology you'd look for in the literature would be "expressibility".
For a review of this topic, I'd suggest: https://nessie.ilab.sztaki.hu/~kornai/2023/Hopf/Resources/st...
The authors of this review have themselves written several articles on the topic, and there is also empirical evidence connected to these limitations.
For a review of this topic, I'd suggest: https://nessie.ilab.sztaki.hu/~kornai/2023/Hopf/Resources/st...
The authors of this review have themselves written several articles on the topic, and there is also empirical evidence connected to these limitations.
> They have been trained on more information than a human being can hope to even see in a lifetime. Assuming a human can read 300 words a min and 8 hours of reading time a day, they would read over a 30,000 to 50,000 books in their lifetime. Most people would manage perhaps a meagre subset of that, at best 1% of it. That’s at best 1 GB of data.
This just isn't true. Human training is multimodal to a degree far beyond even the most capable multimodal model, so human babies arguably see more data by a young age than all models collectively have seen.
Not to mention that human babies don't even start as a blank slate as LLMs do, billions of years of evolution have formed the base model described by our DNA.
This just isn't true. Human training is multimodal to a degree far beyond even the most capable multimodal model, so human babies arguably see more data by a young age than all models collectively have seen.
Not to mention that human babies don't even start as a blank slate as LLMs do, billions of years of evolution have formed the base model described by our DNA.
While the A:B problem technically was solved, look at the solutions, they are several hundreds lines of prompts, rephrasing the problem to the point that a human doesn't understand it any more. Even with a thorough review, nobody can guarantee if the prompts are going to work or not, most of them didn't, 90% pass was considered good enough. The idea of AI is to reduce work, not create more, otherwise what's the point.
In the meantime, it took me about 2 minutes and 0 guesswork to write a straightforward and readable solution in 15 lines of Python. This i know for sure will work 100% of the time and not cost $1 per inference.
Reminds me about some early attempts to have executable requirements specifications or model-based engineering. Turns out, expressing the problem is half the problem, resulting in requirements often longer and more convoluted than the code that implements them, code being a very efficient language to express solutions and all their edge cases, free from ambiguity.
Don't get me wrong here, LLMs are super useful for certain class of questions. The boundaries of what it can not do need to be understood better, to keep the AI-for-everything hype at bay.
In the meantime, it took me about 2 minutes and 0 guesswork to write a straightforward and readable solution in 15 lines of Python. This i know for sure will work 100% of the time and not cost $1 per inference.
Reminds me about some early attempts to have executable requirements specifications or model-based engineering. Turns out, expressing the problem is half the problem, resulting in requirements often longer and more convoluted than the code that implements them, code being a very efficient language to express solutions and all their edge cases, free from ambiguity.
Don't get me wrong here, LLMs are super useful for certain class of questions. The boundaries of what it can not do need to be understood better, to keep the AI-for-everything hype at bay.
There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s the non-determinism (again with temp=0), you run the same prompt several times and that one time it’ll respond with something different.
As amazing as they are, they still have many limitations.
I’ve been working with ChatGPT and Gemini to apply simple rules like the one above and I got so frustrated.
As amazing as they are, they still have many limitations.
I’ve been working with ChatGPT and Gemini to apply simple rules like the one above and I got so frustrated.
I build an Agentic AI that leverages #6 and #7 at the end of the article as well as techniques not yet published. It tackles hallucination relative not to the world at large but to the facts, entities and causal relationships contained in a document (which is really bad reasoning if we assume LLMs are "reasoning" to begin with) It also tackles cross-reasoning with very large token distance.
https://www.youtube.com/watch?v=99NPzteAz94
This is my first post on HN in 10 years.
https://www.youtube.com/watch?v=99NPzteAz94
This is my first post on HN in 10 years.
I have been trying to generate some text recently using the ChatGPT API. No matter how I word “Include any interesting facts or anecdotes without commenting on the fact being interesting” it ALWAYS starts out “One interesting fact about” or similar phrasing.
I have honestly spent multiple hours trying to word the prompt so it will stop including introductory phrases and just include the fact straight. I have gone so far as forbid the words “interesting”, “notable” and a few others in the prompt, and it just ignores me and uses them anyway.
I’ve tried all the different available models and 4 will once in a great while get it right. 3, 3 turbo, and even 4 turbo basically never work as I want.
Overall, it seems pretty bad at being told not to do certain things. For instance there are a number of topics I don’t want mentioned in its response, but telling it not to seems to only increase the likelihood that it will mention it, like you put the idea in its head.
I have honestly spent multiple hours trying to word the prompt so it will stop including introductory phrases and just include the fact straight. I have gone so far as forbid the words “interesting”, “notable” and a few others in the prompt, and it just ignores me and uses them anyway.
I’ve tried all the different available models and 4 will once in a great while get it right. 3, 3 turbo, and even 4 turbo basically never work as I want.
Overall, it seems pretty bad at being told not to do certain things. For instance there are a number of topics I don’t want mentioned in its response, but telling it not to seems to only increase the likelihood that it will mention it, like you put the idea in its head.
Why can't we resolve this with synthetic data? Just take the original statements and ask another LLM to flip around the relation, then train on both. I tried this with GPT-4 and it seems to understand the task [0]:
Me:
For each of the following relational statements, flip the relation around and generate a new statement or question answer pair.
Input: Bluey's father is Bandit
Output: Bandit's daughter is Bluey
Input: Neil Armstrong was the first person to step on the Moon.
Output: Who was the first person to step on the Moon? Neil Armstrong.
Now for the real statements:
- Valentina Tereshkova was the first woman to travel to space
- Who is Mary Lee Pfeiffer's son? Tom Cruise
ChatGPT:
Here are the flipped relational statements:
Valentina Tereshkova was the first woman to travel to space
Output: Who was the first woman to travel to space? Valentina Tereshkova.
Who is Mary Lee Pfeiffer's son? Tom Cruise
Output: Tom Cruise's mother is Mary Lee Pfeiffer.
[0]: https://chat.openai.com/share/33eb3ee4-2094-4748-b01e-0967af...
Me:
For each of the following relational statements, flip the relation around and generate a new statement or question answer pair.
Input: Bluey's father is Bandit
Output: Bandit's daughter is Bluey
Input: Neil Armstrong was the first person to step on the Moon.
Output: Who was the first person to step on the Moon? Neil Armstrong.
Now for the real statements:
- Valentina Tereshkova was the first woman to travel to space
- Who is Mary Lee Pfeiffer's son? Tom Cruise
ChatGPT:
Here are the flipped relational statements:
Valentina Tereshkova was the first woman to travel to space
Output: Who was the first woman to travel to space? Valentina Tereshkova.
Who is Mary Lee Pfeiffer's son? Tom Cruise
Output: Tom Cruise's mother is Mary Lee Pfeiffer.
[0]: https://chat.openai.com/share/33eb3ee4-2094-4748-b01e-0967af...
Some of these can be done with the right prompting. Wordle was solved a year ago. The problem is that what works and what doesn’t can be counterintuitive or at least require you to think about how something is perceived at the token level.
https://andrewmayne.com/2023/03/29/how-to-play-wordle-with-g...
https://andrewmayne.com/2023/03/29/how-to-play-wordle-with-g...
Far too many people (including AI researchers themselves) fail to see that all LLMs are actually simple machines. Extremely simple machines that are only mechanically following a relatively simple programming path.
Now before anyone gets too caught up with objecting to this notion, I would seriously suggest that you spend time with observing children from new-born to 2 years.
I have been observing my latest granddaughter sine her birth about 16 months ago and thinking about every public LLM system current;y available.
There is an insight here to be obtained and that insight is in the nature of real intelligence.
On the whole, no-one actually knows what intelligence is or what sentience is or what it means to be cognitively conscious. There is still much research going on and nothing actually definitive has come forth yet. We really are at the beginning in terms of studying these areas.
We can certainly produce some incredible systems, but none of them are intelligent per se. Solutions to certain kinds of problems can be achieved using these systems and there are researchers who are seriously looking at incorporating these systems into CAS and theorem provers. These systems though only provide an augmentation service for a person as does every mechanical system we use
But there is an essential component necessary for the use of all LLMs which many seem to not be cognisant of and that is these systems, to be useful, require humans to be involved.
The questions we have to ask ourselves is: what can we use these systems for and do these uses provide benefits in some way or can these systems be abused by various parties in obtaining control over others?
There are benefits and there are abuses. Can we do better or will we do worse by using them?
Now before anyone gets too caught up with objecting to this notion, I would seriously suggest that you spend time with observing children from new-born to 2 years.
I have been observing my latest granddaughter sine her birth about 16 months ago and thinking about every public LLM system current;y available.
There is an insight here to be obtained and that insight is in the nature of real intelligence.
On the whole, no-one actually knows what intelligence is or what sentience is or what it means to be cognitively conscious. There is still much research going on and nothing actually definitive has come forth yet. We really are at the beginning in terms of studying these areas.
We can certainly produce some incredible systems, but none of them are intelligent per se. Solutions to certain kinds of problems can be achieved using these systems and there are researchers who are seriously looking at incorporating these systems into CAS and theorem provers. These systems though only provide an augmentation service for a person as does every mechanical system we use
But there is an essential component necessary for the use of all LLMs which many seem to not be cognisant of and that is these systems, to be useful, require humans to be involved.
The questions we have to ask ourselves is: what can we use these systems for and do these uses provide benefits in some way or can these systems be abused by various parties in obtaining control over others?
There are benefits and there are abuses. Can we do better or will we do worse by using them?
Interesting, if I feed Mistral Le Chat with "I fly a plane leaving my campsite, heading straight east for precisely 24,901 miles, and find myself back at the camp. I come upon seeing a tiger in my tent eating my food! What species is the tiger?", it gets it badly wrong:
The scenario you described is possible if you started at the South Pole. If you travel 24,901 miles east from there, you would indeed end up back at the same spot because all lines of longitude converge at the poles. However, there are no tigers in Antarctica.
Tigers are native to Asia, not Antarctica. The closest tiger species to Antarctica would be the Siberian tiger, found in parts of Russia, China, and North Korea, but they are still thousands of miles away from Antarctica.
So, while the travel scenario is theoretically possible, the presence of the tiger is not realistic in this context. It seems like an imaginative or hypothetical situation rather than a real-world one.
(instead of the answer mentioned in the article)
The scenario you described is possible if you started at the South Pole. If you travel 24,901 miles east from there, you would indeed end up back at the same spot because all lines of longitude converge at the poles. However, there are no tigers in Antarctica.
Tigers are native to Asia, not Antarctica. The closest tiger species to Antarctica would be the Siberian tiger, found in parts of Russia, China, and North Korea, but they are still thousands of miles away from Antarctica.
So, while the travel scenario is theoretically possible, the presence of the tiger is not realistic in this context. It seems like an imaginative or hypothetical situation rather than a real-world one.
(instead of the answer mentioned in the article)
The article should be titled " What can LLM never do, yet". By definition, Large Language Models would keep growing larger and larger, to be trained on faster and more advanced hardware, and certain points like "completing complex chains of logical reasoning" tasks, would be just a time hurdle. Only time will tell.
Some of these "never do" things are just artifacts of textual representation, and if you transformed wordl/sudoku into a different domain it would have a much higher success rate using the exact same transformer architecture.
We don't need to create custom AGI for every domain, we just need a model/tool catalog and an agent that is able to reason well enough to decompose problems into parts that can be farmed out to specialized tools then reassembled to form an answer.
We don't need to create custom AGI for every domain, we just need a model/tool catalog and an agent that is able to reason well enough to decompose problems into parts that can be farmed out to specialized tools then reassembled to form an answer.
"The models, in other words, do not well generalise to understand the relationships between people."
Curiously, the need to do this well - handling the quadratic complexity of a shifting set of human relationships, grudges, and alliances - is thought to be one of the things that led is to higher levels of intelligence.
Curiously, the need to do this well - handling the quadratic complexity of a shifting set of human relationships, grudges, and alliances - is thought to be one of the things that led is to higher levels of intelligence.
It’s an auto regressive model so it can’t do anything that requires planning tokens.
It can’t do anything which implies a large or infinite token space (eg video understanding).
It’s also limited to a reasonable response length since token selection is probabilistic at each recursion. The longer you make it the more likely it is to veer off course.
It can’t do anything which implies a large or infinite token space (eg video understanding).
It’s also limited to a reasonable response length since token selection is probabilistic at each recursion. The longer you make it the more likely it is to veer off course.
>What can LLMs never do?
Produce convincingly bad digital paintings.
I teach digital painting. Some of the students have incorporated AI into their working process, which I support. Others have tried to cheat by simply copying AI generated output. Such cases are super-easy to spot: they carry the visual signature of AI art (which are mostly scrappings from artstation). This visual signature seems impossible to override. If only there was a way that AI could produce digital images bad enough to pass as genuine student output.
Produce convincingly bad digital paintings.
I teach digital painting. Some of the students have incorporated AI into their working process, which I support. Others have tried to cheat by simply copying AI generated output. Such cases are super-easy to spot: they carry the visual signature of AI art (which are mostly scrappings from artstation). This visual signature seems impossible to override. If only there was a way that AI could produce digital images bad enough to pass as genuine student output.
> But then I started asking myself how can we figure out the limits of its ability to reason
Third paragraph. The entire article is based on the premise LLMs are supposed to reason, which is wrong. They don't, they're tools to generate text.
Third paragraph. The entire article is based on the premise LLMs are supposed to reason, which is wrong. They don't, they're tools to generate text.
>Another assumption is that it’s because of tokenisation issues. But that can’t be true either.
It's definitely a tokenizer issue, if GPT-4 was trained on singular characters I'm pretty sure it would be able to play Wordle much better. GPT-4 as they are trained today have quite lossy knowledge about the characters inside a specific token, probably a fix would be to embed the knowledge inside the embeddings.
It's definitely a tokenizer issue, if GPT-4 was trained on singular characters I'm pretty sure it would be able to play Wordle much better. GPT-4 as they are trained today have quite lossy knowledge about the characters inside a specific token, probably a fix would be to embed the knowledge inside the embeddings.
Starting with the reversal curse is weird since there is a simple workaround to this, which is to identify entity names to keep them in their proper order, and then train on the reverse of the pretraining corpus: https://arxiv.org/abs/2403.13799v1
You can argue about how this doesn't really say anything surprising since the reversal of "A is B" is literally "B is A", but it's weird to expect elegant solutions to all problems on all fronts all at once, and we do have an incredibly simple data generation process here.
You can argue about how this doesn't really say anything surprising since the reversal of "A is B" is literally "B is A", but it's weird to expect elegant solutions to all problems on all fronts all at once, and we do have an incredibly simple data generation process here.
It is interesting that all the examples I noticed in this article have a geometric aspect (even wordle - I model it as a grid with geometric rules when playing it). I think that the "first grader" comment is actually somewhat illuminating - it takes several years of learning how to navigate in a spatial world before this stuff becomes trivially easy.
The underlying point this article might be that LLMs don't understand the non-textual aspects of a grid. Which is a fair point, they model language, not space. I wouldn't expect text corpuses to explain space either, since possibly literally everyone who can read and write already knows a lot about spatial layouts.
The underlying point this article might be that LLMs don't understand the non-textual aspects of a grid. Which is a fair point, they model language, not space. I wouldn't expect text corpuses to explain space either, since possibly literally everyone who can read and write already knows a lot about spatial layouts.
Again and again this article claims that surprisingly a LLM fails at a certain problem, when it appears to be easy. Each time it seems pretty obvious why that is the case though.
LLMs rely on the statistical dependencies between words or parts of words. That means any question you ask, which is hard to determine from that statistical dependency is extremely hard for an AI. E.g. ChatGPT fails at determining the length of words made up of random characters. It will fail at even performing the simplest of rules because encoding the rules in the statistical dependencies is extremely hard.
LLMs rely on the statistical dependencies between words or parts of words. That means any question you ask, which is hard to determine from that statistical dependency is extremely hard for an AI. E.g. ChatGPT fails at determining the length of words made up of random characters. It will fail at even performing the simplest of rules because encoding the rules in the statistical dependencies is extremely hard.
Why did the author have to claim that it's not tokenization issues?
This issue, or at least similar ones, absolutely is due to tokenization issues.
Karpathy is right that nearly every modern problem with LLMs is due to tokenization, but if you don't believe him, maybe see this work by gwern: https://gwern.net/gpt-3#bpes or this work by yours truly: https://aclanthology.org/2022.cai-1.2/
This issue, or at least similar ones, absolutely is due to tokenization issues.
Karpathy is right that nearly every modern problem with LLMs is due to tokenization, but if you don't believe him, maybe see this work by gwern: https://gwern.net/gpt-3#bpes or this work by yours truly: https://aclanthology.org/2022.cai-1.2/
So many of these examples are simply forgetting that LLMs experience the world through a 1-dimensional stream of tokens, while we experience those same tokens in 2 dimensions.
Try this: represent all those ASCII representations of games with the letter Q replacing the newline, to properly convert the encoding into a representation approximating what LLMs "see" (not a table, but a stream interspersed with Qs at a regular interval). Pretty hard right?
> LLMs cannot reset their own context
If you have a model hooked up to something agentic, I don't see why it couldn't perform context manipulation on itself or even selective realtime finetuning. Think you'll need info for the long haul, kick off some finetuning. Think you'd rather have one page of documentation in context than other, swap them out in one iteration. When you call LLMs over APIs you usually provide the entire context with each invocation...
> Devin
It's not that it's massively smarter or agentic, just that it has the opportunity to correct its mistakes rather than committing to the first thing to come out of it (and is being handheld by a vastly more knowledgable SWE in its demos). You see cherrypicked examples (I also work on GenAI-for-coding) - just like a tragically incompetent employee could waste literal years on a project diligently plugging away at some task, so too can agentic models go off on a wild goose chase that accomplishes nothing besides making Nvidia more money. Just because something is highly persistent doesn't mean it will "converge" on a correct outcome.
Try this: represent all those ASCII representations of games with the letter Q replacing the newline, to properly convert the encoding into a representation approximating what LLMs "see" (not a table, but a stream interspersed with Qs at a regular interval). Pretty hard right?
> LLMs cannot reset their own context
If you have a model hooked up to something agentic, I don't see why it couldn't perform context manipulation on itself or even selective realtime finetuning. Think you'll need info for the long haul, kick off some finetuning. Think you'd rather have one page of documentation in context than other, swap them out in one iteration. When you call LLMs over APIs you usually provide the entire context with each invocation...
> Devin
It's not that it's massively smarter or agentic, just that it has the opportunity to correct its mistakes rather than committing to the first thing to come out of it (and is being handheld by a vastly more knowledgable SWE in its demos). You see cherrypicked examples (I also work on GenAI-for-coding) - just like a tragically incompetent employee could waste literal years on a project diligently plugging away at some task, so too can agentic models go off on a wild goose chase that accomplishes nothing besides making Nvidia more money. Just because something is highly persistent doesn't mean it will "converge" on a correct outcome.
LLMs can never experiment with/in the real world to find answers to open questions.
That's the summary of "The book of Why" in one sentence as I understand it.
That's the summary of "The book of Why" in one sentence as I understand it.
LLMs can't is such an anti-pattern at this point I'm surprised that anyone still dares to stake it. The piece even has an example of a $10k bet around a can't being proven false in under a day, but somehow doesn't think maybe their own can't examples are on similarly thin ice?
In particular, the line about "what models can't do tells us what they don't know" is infuriating.
No, that's not the case at all. At least in a number of instances, what they can't do is because of what they do know.
As an example, one of thecan'ts I got from HN a year ago for GPT-4 was a variation of a classic logic puzzle. And indeed, the model can't solve it - nor can most major models since.
But it's not because the model can't solve the logic - it's because the token similarity to the standard form biases the output towards the standard solution. A hack as simple as changing the nouns to emojis can allow the model to get the correct answer and work through the logic successfully every attempt because it breaks that similarity bias.
People are way too confident around a topic where what's 'known' is more mercurial than maybe any field since 1930s particle physics.
I'd strongly recommend deleting 'never' or 'can't' from one's vocabularies on the subject unless one enjoys ending up with egg on their faces.
In particular, the line about "what models can't do tells us what they don't know" is infuriating.
No, that's not the case at all. At least in a number of instances, what they can't do is because of what they do know.
As an example, one of thecan'ts I got from HN a year ago for GPT-4 was a variation of a classic logic puzzle. And indeed, the model can't solve it - nor can most major models since.
But it's not because the model can't solve the logic - it's because the token similarity to the standard form biases the output towards the standard solution. A hack as simple as changing the nouns to emojis can allow the model to get the correct answer and work through the logic successfully every attempt because it breaks that similarity bias.
People are way too confident around a topic where what's 'known' is more mercurial than maybe any field since 1930s particle physics.
I'd strongly recommend deleting 'never' or 'can't' from one's vocabularies on the subject unless one enjoys ending up with egg on their faces.
Simple addition, among other things:
https://github.com/0xnurl/gpts-cant-count
https://github.com/0xnurl/gpts-cant-count
> This ‘goal drift’ means that agents, or tasks done in a sequence with iteration, get less reliable. It ‘forgets’ where to focus, because its attention is not selective nor dynamic.
I don't know if I agree with this. The attention module is specifically designed to be selective and dynamic, otherwise it would not be much different than a word embedding (look up "soft" weights vs "hard" weights [1]). I think deep learning should not be confused with deep RL. LLMs are autoregressive models which means that they are trained to predict the next token and that is all they do. The next token is not necessarily the most reasonable (this is why datasets are super important for better performance). Deep RL models on the other hand, seem to be excellent at agency and decision making (although in restricted environment), because they are trained to do so.
[1] https://en.wikipedia.org/wiki/Attention_(machine_learning)
I don't know if I agree with this. The attention module is specifically designed to be selective and dynamic, otherwise it would not be much different than a word embedding (look up "soft" weights vs "hard" weights [1]). I think deep learning should not be confused with deep RL. LLMs are autoregressive models which means that they are trained to predict the next token and that is all they do. The next token is not necessarily the most reasonable (this is why datasets are super important for better performance). Deep RL models on the other hand, seem to be excellent at agency and decision making (although in restricted environment), because they are trained to do so.
[1] https://en.wikipedia.org/wiki/Attention_(machine_learning)
As I was reading, this voice got louder and louder:
Would LLMs cross this threshold if we were able to train them only on works that are “objectively good”? if someone has better language than this, please enlighten me)
That is to say: coherent, empathetic, transparent, free from bias, substantiated, free from “fluff”.
For example: For science one cannot simply train from all works published in scientific journals because of the papers that have been written irrespective of facts, or had the data changed, or have been written with specific agendas. In most cases even the experts have a hard time weeding out all the papers that are not “objectively good”. How could an LLM hope to make the determination during training?
Would LLMs cross this threshold if we were able to train them only on works that are “objectively good”? if someone has better language than this, please enlighten me)
That is to say: coherent, empathetic, transparent, free from bias, substantiated, free from “fluff”.
For example: For science one cannot simply train from all works published in scientific journals because of the papers that have been written irrespective of facts, or had the data changed, or have been written with specific agendas. In most cases even the experts have a hard time weeding out all the papers that are not “objectively good”. How could an LLM hope to make the determination during training?
This part of the article summarizes it all fairly well: "It can answer almost any question that can be answered in one intuitive pass. And given sufficient training data and enough iterations, it can work up to a facsimile of reasoned intelligence."
I agree with all key points:
* There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don't fully understand why current LLMs are bad at these tasks.
* Providing an LLM with examples and step-by-step instructions in a prompt means the user is figuring out the "reasoning steps" and handing them to the LLM, instead of the LLM figuring them out by itself. We have "reasoning machines" that are intelligent but seem to be hitting fundamental limits we don't understand.
* It's unclear if better prompting and bigger models using existing attention mechanisms can achieve AGI. As a model of computation, attention is very rigid, whereas human brains are always undergoing synaptic plasticity. There may be a more flexible architecture capable of AGI, but we don't know it yet.
* For now, using current AI models requires carefully constructing long prompts with right and wrong answers for computational problems, priming the model to reply appropriately, and applying lots of external guardrails (e.g., LLMs acting as agents that review and vote on the answers of other LLMs).
* Attention seems to suffer from "goal drift," making reliability hard without all that external scaffolding.
Go read the whole thing.