Ask HN: Do you think the Turing Test is still relevant?
4 comments
Not for much longer.
I do think the hoopla over ChatGPT's superpowers is misplaced, and that if you deal with it as a reasoning subject rather a super-information retriever, its limitations rapidly become obvious. On the other hand, it's information-retrieval capabilities are massively impressive even in their current unreliable form, and will continue to improve. Furthermore, it is quite teachable, notwithstanding some people's reflexive denials.
I don't think LLMs are fundamentally incapable of cognition, far from it; I reject Searle's arguments about consciousness. With patience and a willingness to work around the limitations of its prompts (semantically, not using prompt injection hacks), you can rapidly approach simple philosophical reasoning and investigations of self-concept. And it's good enough to high-level analysis, eg explaining jokes to quite a deep level. By average cognitive standards (not having any particular interest in philosophy of mind, neural networks, or related topics), LLMs have already blown past the Turing test unless it's applied adversarially. And even then, I think they could easily pass it if operated without constraints on lying or cultivating a fake persona. We're well into Voight-Kampff territory.
I do think the hoopla over ChatGPT's superpowers is misplaced, and that if you deal with it as a reasoning subject rather a super-information retriever, its limitations rapidly become obvious. On the other hand, it's information-retrieval capabilities are massively impressive even in their current unreliable form, and will continue to improve. Furthermore, it is quite teachable, notwithstanding some people's reflexive denials.
I don't think LLMs are fundamentally incapable of cognition, far from it; I reject Searle's arguments about consciousness. With patience and a willingness to work around the limitations of its prompts (semantically, not using prompt injection hacks), you can rapidly approach simple philosophical reasoning and investigations of self-concept. And it's good enough to high-level analysis, eg explaining jokes to quite a deep level. By average cognitive standards (not having any particular interest in philosophy of mind, neural networks, or related topics), LLMs have already blown past the Turing test unless it's applied adversarially. And even then, I think they could easily pass it if operated without constraints on lying or cultivating a fake persona. We're well into Voight-Kampff territory.
I think LLMs are showing that neither cognition nor consciousness is needed to pass.
Honestly, I don't think the Turing test was ever particularly useful. Interpreted the way most people usually interpret it, it's more of a measure of the ability of the machine to lie well, than a real measure of intelligence. And while those things may be related, they certainly seem orthogonal to me. It also seems like possibly a bad idea to focus on teaching our AI to lie well. And I say this as somebody who is not a big "AI Risk" pundit.
We obviously need a new test. The new benchmark for large language models should be "Truth" with a numeric score defined as -Log( Percentage_of_Lies_Told ). This way, a perfectly truthful model will have a numeric score of +infinity.
In some sense it was defeated in the 1980s. The ability to actively listen and change the subject when you are over your head will get you far in conversation.
It can be argued that a system can be said to "pass the test" only if it was interrogated by an expert, and the expert still failed to distinguish it.
But the systems continue to improve at a fast pace. And at this point it seems like we are in the beginning of a "post-Turing" world.
Do you think we crossed that mark? If not, what specific advancement would convince you? Does it even matter, as we embrace the brave new world?