Ask HN: Why is GPT-4 or Claude-2 so bad at tic-tac-toe?
5 comments
I agree with jqpabc123 except for one point: that it cannot 'reason' and it is not smart. I disagree with this.
The reason it cannot play well is because it has very little 'experience' (training data) with it. It's been trained on 'what the game is', it has not been trained how to win.
You can think of it a bit like driving. You can know what driving is, but it doesn't make you a driver if you've never driven before.
You can ask a genius who's never played before to play tic tac toe with you, tell it the rules, they will likely not win on the first attempt or play optimally. This doesn't mean that person isn't a genius.
You said humans are able to 'learn' to play it rapidly. So is GPT, in training mode, it can process a million games in seconds, where a human can't.
The problem here, is it simply has no experience.
If I told you every time you played tic tac toe against me, you would forget all your experience the next time we played, would you play optimally?
The reason it cannot play well is because it has very little 'experience' (training data) with it. It's been trained on 'what the game is', it has not been trained how to win.
You can think of it a bit like driving. You can know what driving is, but it doesn't make you a driver if you've never driven before.
You can ask a genius who's never played before to play tic tac toe with you, tell it the rules, they will likely not win on the first attempt or play optimally. This doesn't mean that person isn't a genius.
You said humans are able to 'learn' to play it rapidly. So is GPT, in training mode, it can process a million games in seconds, where a human can't.
The problem here, is it simply has no experience.
If I told you every time you played tic tac toe against me, you would forget all your experience the next time we played, would you play optimally?
can process a million games in seconds, where a human can't.
A human doesn't need to. A human can apply reason and logic and at least play the game given only a few simple rules.
GPT has poor reasoning ability and doesn't comprehend the rules --- so it relies on database lookup to sorta fake it. Without this, it's kinda helpless. It clearly lacks any real comprehension --- aka "intelligence".
Using statistics, it can fabricate text that "sounds" plausible but it has no comprehension of the correctness (or lack there of) of what it produces.
A human doesn't need to. A human can apply reason and logic and at least play the game given only a few simple rules.
GPT has poor reasoning ability and doesn't comprehend the rules --- so it relies on database lookup to sorta fake it. Without this, it's kinda helpless. It clearly lacks any real comprehension --- aka "intelligence".
Using statistics, it can fabricate text that "sounds" plausible but it has no comprehension of the correctness (or lack there of) of what it produces.
But is that really the case? It's actually not just a database lookup. It is abstracting and compressing throughout the training process.
And it clearly learns certain rules about our world. It seems LLMs can reason, but reasoning is a big field and it hasn't mastered all it's forms, i.e. search future scenarios or spatial reasoning.
And it clearly learns certain rules about our world. It seems LLMs can reason, but reasoning is a big field and it hasn't mastered all it's forms, i.e. search future scenarios or spatial reasoning.
I think this paper has an interesting way of training problem solving:
https://venturebeat.com/ai/microsoft-unveils-lema-a-revoluti...
I submitted to HN but nobody seemed to care:
https://news.ycombinator.com/item?id=38128012
I looks like it basically uses GPT-4 to train a smaller model on problem solving.
https://venturebeat.com/ai/microsoft-unveils-lema-a-revoluti...
I submitted to HN but nobody seemed to care:
https://news.ycombinator.com/item?id=38128012
I looks like it basically uses GPT-4 to train a smaller model on problem solving.
But after trying for 2h to get it work (even with GPT-4V) it seems like a fundamental limitation.
It obviously hasn't been "trained" for tic-tac-toe. The way to train it is using statistics --- present every possible position and the correct response so it can build a database.
There is no logic or reasoning involved --- it's all statistics. It's not what we call "smart". Any ability to "reason" is just a statistical illusion.
It obviously hasn't been "trained" for tic-tac-toe. The way to train it is using statistics --- present every possible position and the correct response so it can build a database.
There is no logic or reasoning involved --- it's all statistics. It's not what we call "smart". Any ability to "reason" is just a statistical illusion.
The Turing Test sounds cool, and "is he a clever conversationalist?" is a fairly good test - of social intelligence and class, for casual use in human society.
But current "AI's" are intelligent kinda like pocket calculators are intelligent.
But current "AI's" are intelligent kinda like pocket calculators are intelligent.
The New Yorker wrote a nice article on LLMs and how they work, "What Kind of Mind Does ChatGPT Have?"
But after trying for 2h to get it work (even with GPT-4V) it seems like a fundamental limitation.
I've found a HN submission [1] where someone used a brute-force prompt to get it to play correctly, but as the top/only comment points out, it's a limited action space and enumerating most of it seems moot.
I was hoping for a more reasonable prompt. After all humans are able learn tic-tac-toe rapidly.
current hypotheses:
1) tic-tac-toe requires "spatial reasoning" and LLMs train on sequences (somehow GPT-4V didn't elevate that constraint)
2) tic-tac-toe requires "search" of future scenarios
Would love to hear what you think/know!
---
Previous discussion about T3 and GPT-4: https://news.ycombinator.com/item?id=35216614 (7 months ago)
[1] https://news.ycombinator.com/item?id=37626918