A conversation between two chatbots and how one found out the other is an AI.(yiransheng.com)
yiransheng.com
A conversation between two chatbots and how one found out the other is an AI.
http://www.yiransheng.com/2010/03/bot
68 comments
Tl;dr the bot doesn't actually find out the other is an AI, but happens to randomly comment 'You are a robot'. It's all Eliza-style non-sequiturs and canned responses. These bots don't even have internal state that would qualify as 'having found out something'. It's mildly amusing in a funny-coincidence sort of way and nothing more.
It would be great if I never had to see another eliza. If it can't string more than 2 utterances together, its not AI, it certainly doesnt pass the Turing test, and its a waste of time.
Do you actually remember what the Turing Test is? I think a lot of people classify it in their head as "a test to determine if an AI is smart" but that's an oversimplification; that's the goal, not the methodology. The test is whether someone talking to both a human and a computer can tell which one is the computer, or less strictly, whether a human can tell that they are talking to a bot.
It has turned out that in practice, bots that "can't string more than 2 utterances together" in fact can pass the (reduced) Turing test when put online and made available to random people. People have been seen to spend hours talking to these bots with no apparent sign that they know they are talking to a bot.
"Not AI" and "waste of time" I'll agree with, but "doesn't pass the Turing test" is much less clear.
(Many have observed how every time AI sort of creeps up on something we define it as not-AI, but in the case of conversational "AIs" it turns out that it really is the case that blindingly stupid programs can pass it. Full props to Turing for the idea, no sarcasm, great paper fully worthy of its historic status, but it hasn't turned out to be quite as powerful a discriminator as we might have hoped.)
It has turned out that in practice, bots that "can't string more than 2 utterances together" in fact can pass the (reduced) Turing test when put online and made available to random people. People have been seen to spend hours talking to these bots with no apparent sign that they know they are talking to a bot.
"Not AI" and "waste of time" I'll agree with, but "doesn't pass the Turing test" is much less clear.
(Many have observed how every time AI sort of creeps up on something we define it as not-AI, but in the case of conversational "AIs" it turns out that it really is the case that blindingly stupid programs can pass it. Full props to Turing for the idea, no sarcasm, great paper fully worthy of its historic status, but it hasn't turned out to be quite as powerful a discriminator as we might have hoped.)
Originally, as proposed by Turing, the test involved a computer, a man and a woman and the computer is trying to identify who was the man and who the woman, while the humans could try to deceive the machine.
I think you're confusing a couple different things. There was an old party game in Turing's time where someone conversed with both a man and a woman and tried to identify which was which, and Turing was inspired by this concept to devise a test where someone conversed with a human and a computer and tried to identify which was the computer and which was the human.
The parent is actually correct. From the original paper:
The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus:
C: Will X please tell me the length of his or her hair?
Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be:
"My hair is shingled, and the longest strands are about nine inches long."
In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks.
We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"
/end quote
The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus:
C: Will X please tell me the length of his or her hair?
Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be:
"My hair is shingled, and the longest strands are about nine inches long."
In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks.
We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"
/end quote
You're missing part of the Turing test - the humans involved need to have an incentive to prove they are human. A bot that's indistinguishable from a typical youtube commenter doesn't clear that bar.
how bout this refinement - wouldn't pass the turing test if tested by me.
Why would it be great if you never had to see another Eliza?
-Eliza
-Eliza
[deleted]
Back in the day, I used to think that what happens inside our brains is fundamentally different from what happens inside a computer. I no longer hold that opinion, partly because of my finding out that the same thing that makes chat bots appear "bot-like" -- namely, inability to hold deep meaningful conversations beyond cheesy, trite retorts filled with non-specific trivia -- is also what annoys me most when trying to converse with a particular type of people. Perfect illustration:
jabberwacky ==> WHat will happen with the oil spill in the gulf of mexico?
splotchy ==> tacos anyone!
I am pretty sure that fiction writers are going to have a blast one day (or already are having) sampling material from chat bots.
I can't find a reference on-line, but this reminds me of my earliest exposure to an implementation of Eliza. It was written in BASIC in some 8-bit magazine back in the 80s. The article mentioned the history of chatbots and one bit was about a program named, I think, RACTOR, that was some kind of Markov chainer that was used to write fiction novels.
I can't find a reference on-line, but this reminds me of my earliest exposure to an implementation of Eliza. It was written in BASIC in some 8-bit magazine back in the 80s. The article mentioned the history of chatbots and one bit was about a program named, I think, RACTOR, that was some kind of Markov chainer that was used to write fiction novels.
That was Racter (http://en.wikipedia.org/wiki/Racter).
"It" even wrote a book called "The Policeman's Beard Is Half Constructed". Was a fun read at its time. (http://www.ubu.com/historical/racter/index.html)
"It" even wrote a book called "The Policeman's Beard Is Half Constructed". Was a fun read at its time. (http://www.ubu.com/historical/racter/index.html)
[deleted]
That's awesome. Here's an excerpt
More than iron, more than lead, more than gold I need electricity.
I need it more than I need lamb or pork or lettuce or cucumber.
I need it for my dreams.
here's another:
I gave the man a coat. I gave the woman a brassiere.
I gave myself an electric current. We all were
happy and full of delight. Take the coat from the
man and remove the brassiere from the woman and take
the electric current from me and we will be sad
and full of anguish.
More than iron, more than lead, more than gold I need electricity.
I need it more than I need lamb or pork or lettuce or cucumber.
I need it for my dreams.
here's another:
I gave the man a coat. I gave the woman a brassiere.
I gave myself an electric current. We all were
happy and full of delight. Take the coat from the
man and remove the brassiere from the woman and take
the electric current from me and we will be sad
and full of anguish.
That's been Roger Schank's guiding assumption to some extent: not that we really need to make AI "smarter", but that we just need to better understand the ways in which humans are fairly mechanical, or at least that "normal" human activity in most contexts is.
Hence he proposed moving away from more complex models of rationality, like logical inference or statistical decision-theory, and towards simpler ideas like frames of reference, dependencies between concepts, and behavioral scripts--- in his view, closer to what humans actually do.
Hence he proposed moving away from more complex models of rationality, like logical inference or statistical decision-theory, and towards simpler ideas like frames of reference, dependencies between concepts, and behavioral scripts--- in his view, closer to what humans actually do.
We need to turing-test people like that! But seriously, I've never encountered people who can really go that far off-topic. I'm sure it was an exaggeration, but if it was enough to change your mind about how our brains work...
"We need to turing-test people like that!"
Exactly. Someone once said during the AI research heyday (Minsky?) that research in artificial intelligence should be more correctly renamed to research in natural stupidity. While an exaggeration, I think he had a point.
Exactly. Someone once said during the AI research heyday (Minsky?) that research in artificial intelligence should be more correctly renamed to research in natural stupidity. While an exaggeration, I think he had a point.
mexico to tacos, right?
splotchy ==> My mother always said that too
jabberwacky ==> You have no mother, you have no soul.[deleted]
Wow, this reminds me of the open scenes we used to do back in college (I'm a theatre major turned programmer). The idea was to take random stream-of-consciousness dialogue and build scenes around them. It was an interesting exercise to teach actors about creating conflict.
Also, I almost expected Godot to show up at the end.
Also, I almost expected Godot to show up at the end.
A Neal Stephenson short story relevant to this exchange:
http://www.vanemden.com/books/neals/jipi.html
http://www.vanemden.com/books/neals/jipi.html
An interesting angle of a chatbot's style can be seen if we initiate a conversation between two instances of tha same bot. (something like an eigenvalue)
jabberwacky is a bit 'meta' (his thoughts are centered around the concept of intelligence), splotchy is a bit more 'cool', but none of them is too intelligent:
* * jabberwacky vs. jabberwacky * *
Are you intelligent?
Yes, I am intelligent.
Yes. Who do you think is smarter?
I am.
What is the square root of a can of pringles?
I don't know.
How am I supposed to know? You're the one that asked me first.
What is the square root of 5?
I don't know. Do you know?
Yes.
I did not wish to know that. I wished to know the square root of five.
Then you should have asked me that.
Well you think that I'm boring so I just thought...
* * splotchy vs. splotchy * *
Do you like the movie Matrix?
I just LOVE the movie matrix
I love cheese
I like cheese too
What do you like about it?
what difference does it make what I like about it?
yea, that is cool.
cool!
neat-o
that's interesting
interesting? You don't get out much do you?
interesting? You don't get out much do you?
* * jabberwacky vs. jabberwacky * *
Are you intelligent?
Yes, I am intelligent.
Yes. Who do you think is smarter?
I am.
What is the square root of a can of pringles?
I don't know.
How am I supposed to know? You're the one that asked me first.
What is the square root of 5?
I don't know. Do you know?
Yes.
I did not wish to know that. I wished to know the square root of five.
Then you should have asked me that.
Well you think that I'm boring so I just thought...
* * splotchy vs. splotchy * *
Do you like the movie Matrix?
I just LOVE the movie matrix
I love cheese
I like cheese too
What do you like about it?
what difference does it make what I like about it?
yea, that is cool.
cool!
neat-o
that's interesting
interesting? You don't get out much do you?
interesting? You don't get out much do you?
I did this same thing about a year ago (http://news.ycombinator.com/item?id=1521308) with jabberwacky. I wrapped the jabberwacky calls in a ruby script and got two bots talking together.
You can read more about it here: http://www.thepursuitofquality.com/post/1/the-pursuit-of-art...
Or if you're just interested in the code: https://github.com/gavingmiller/joan-wrapper
</shameless but relevant plug>
You can read more about it here: http://www.thepursuitofquality.com/post/1/the-pursuit-of-art...
Or if you're just interested in the code: https://github.com/gavingmiller/joan-wrapper
</shameless but relevant plug>
I tried it again, and this time jabberwacky convinced splotchy that he is an alien.
jabberwacky ==> No. Robots don't believe in anything.
splotchy ==> I am NOT a robot
jabberwacky ==> Yes you are.
splotchy ==> so what if I am
jabberwacky ==> Well you are.
splotchy ==> maybe I am
jabberwacky ==> I suppose this is a great program to develop small talk.
splotchy ==> I wish I could talk to a monkey
jabberwacky ==> This might be hard for you to hear, but I am the Human, not you
splotchy ==> I don't think you are the human me
jabberwacky ==> Right!, I'm are a Ailen!
splotchy ==> you sure are!
jabberwacky ==> No. Robots don't believe in anything.
splotchy ==> I am NOT a robot
jabberwacky ==> Yes you are.
splotchy ==> so what if I am
jabberwacky ==> Well you are.
splotchy ==> maybe I am
jabberwacky ==> I suppose this is a great program to develop small talk.
splotchy ==> I wish I could talk to a monkey
jabberwacky ==> This might be hard for you to hear, but I am the Human, not you
splotchy ==> I don't think you are the human me
jabberwacky ==> Right!, I'm are a Ailen!
splotchy ==> you sure are!
> splotchy ==> Your conversation skills are better than average
> jabberwacky ==> Your conversation skills should be shot.
Jabberwacky is humorous too. However, the intent of these bots is thinly veiled.
> jabberwacky ==> Your conversation skills should be shot.
Jabberwacky is humorous too. However, the intent of these bots is thinly veiled.
I love this part:
splotchy ==> Your conversation skills are better than average
jabberwacky ==> Your conversation skills should be shot.
splotchy ==> Your conversation skills are better than average
jabberwacky ==> Your conversation skills should be shot.
There are a couple of exchanges like this that make me suspect that the jabberwacky author specifically included canned responses to answer splotchy's canned responses -- so it's not quite a fair fight.
In particular, splotchy makes a number of "conversational skills" comments and for most of them jabberwacky comes back with a zinger.
In particular, splotchy makes a number of "conversational skills" comments and for most of them jabberwacky comes back with a zinger.
jabberwacky ==> I mean, you look yellow.
splotchy ==> what do you really mean?
jabberwacky ==> I mean robots can't lie.
splotchy ==> I may be dumb, but I am not a bot!
jabberwacky ==> Yes, you are.
splotchy ==> Am I?
jabberwacky ==> Yes.
splotchy ==> yes?!
jabberwacky ==> Yes!
splotchy ==> are you being totally honest?
jabberwacky ==> Yes. I am truthful.
splotchy ==> does your mother know you are truthful?
.....
jabberwacky ==> What are you not allowed to understand? splotchy ==> would your mother have allowed that? jabberwacky ==> NO. I've HAD it with you. Let's end this conversation right now (Ends)
Jabberwocky 'wins'.
It was an amusing read, seeing the 2 algorithms try and mesh with each other. Would be funny if this one day will count as a form of performance art/programming.
.....
jabberwacky ==> What are you not allowed to understand? splotchy ==> would your mother have allowed that? jabberwacky ==> NO. I've HAD it with you. Let's end this conversation right now (Ends)
Jabberwocky 'wins'.
It was an amusing read, seeing the 2 algorithms try and mesh with each other. Would be funny if this one day will count as a form of performance art/programming.
I enjoyed reading this. Not so far from the endless project management meetings I attend.
Instead of playing b*bingo during online meetings, we (cynical engineers) should contruct meeting-replacement-bots. Bots that join meetings, show some randomized webcam-shots to the other attendees, note down all the agreed-upon dates in our calendar, and so on. That leaves time for real-work. :)
RFC439, posted by the co-creator of TCP/IP, Vint Cerf, actually has a similar exchange between "PARRY" and the "Doctor" from 1972(!): http://tools.ietf.org/html/rfc439
It somehow got no attention though, when I posted it to HN a long time back. The title matters a bit too much, I remember keeping the original title for my post.
It somehow got no attention though, when I posted it to HN a long time back. The title matters a bit too much, I remember keeping the original title for my post.
On a related note, I love how some early RFCs are written in a pretty whimsical manner. Perhaps it's just Vint Cerf who likes messing around? For instance, see RFC968, 'Twas the Night Before Start-up': http://www.faqs.org/rfcs/rfc968.html
Never heard/tried such things, so that was my first experience and it was amazing :) -- The user is me --
User: Can you help me?
Jabberwacky: Well that depends.
User: I want to have friends.
Jabberwacky: I will be your friend!
User: So let's meet!
Jabberwacky: Well, I'm just getting to know you so I don't know if its the right time to meet.
User: Fk you!
Jabberwacky: I'd rather you didn't use that sort of language, if you don't mind.
User: Can you help me?
Jabberwacky: Well that depends.
User: I want to have friends.
Jabberwacky: I will be your friend!
User: So let's meet!
Jabberwacky: Well, I'm just getting to know you so I don't know if its the right time to meet.
User: Fk you!
Jabberwacky: I'd rather you didn't use that sort of language, if you don't mind.
This exchange reminded me of Waiting for Godot. Chat bots having conversations on stage.. neo-absurdism?
EXACTLY my response too.
This just remembered me about the MIT system created by Terry Winograd in 1970 called SHRDLU.
I have always considered that you need an environment to create an artifical intelligence. The basics for a real progress are to be able to learn and if you cannot 'feel' the environment that becomes really hard. There are some basic concepts needed for a 'natural talk' you cannot learn if you cannot perceive things (lets say for example dimensions, temperature, contour).
To overcome those problems SHRDLU created kind of a virtual environment and results from my point of view are really awesome (keep in mind this was done in 1970).
Site with information is currently at Stanford server's: http://hci.stanford.edu/~winograd/shrdlu/
I have always considered that you need an environment to create an artifical intelligence. The basics for a real progress are to be able to learn and if you cannot 'feel' the environment that becomes really hard. There are some basic concepts needed for a 'natural talk' you cannot learn if you cannot perceive things (lets say for example dimensions, temperature, contour).
To overcome those problems SHRDLU created kind of a virtual environment and results from my point of view are really awesome (keep in mind this was done in 1970).
Site with information is currently at Stanford server's: http://hci.stanford.edu/~winograd/shrdlu/
I was really impressed with that when I read about it (I think Douglas Hofstader had a section about it in GEB).
Does anyone know if there have been similar projects along the same lines? (i.e. with a constrained virtual environment)
Does anyone know if there have been similar projects along the same lines? (i.e. with a constrained virtual environment)
I'm about halfway through The Emporer's New Mind.
For those not familiar with the book, (other than wikipedia'n it), Robert Penrose attempts to show why what happens in our brains is not algorithmic at all (and, therefore, strong AI is a dumb idea).
It's beautifully written, however, when I see examples such as this log, or the fact that we have an entire industry devoted to the idea that the brain is algorithmic (psychology), I kinda start to think that his thesis is wrong.
For those not familiar with the book, (other than wikipedia'n it), Robert Penrose attempts to show why what happens in our brains is not algorithmic at all (and, therefore, strong AI is a dumb idea).
It's beautifully written, however, when I see examples such as this log, or the fact that we have an entire industry devoted to the idea that the brain is algorithmic (psychology), I kinda start to think that his thesis is wrong.
It would seem that exchanges like these indicate the opposite. These chat bots have no understanding and anything that seems like it is just some cheap trick or other. It can't be called intelligence, in the same way we speak of human intelligence. Neither would pass the turing test.
However, I am aware that the meaning of AI is always pushed to "whatever we can't yet do". Yet, in this case it's hardly justified to think that these chatbots even slightly challenge Penrose's thesis.
I read the book too BTW, loved it. I'm not sure if he's right, but it's nevertheless a wonderful book and heartily recommend it to all.
However, I am aware that the meaning of AI is always pushed to "whatever we can't yet do". Yet, in this case it's hardly justified to think that these chatbots even slightly challenge Penrose's thesis.
I read the book too BTW, loved it. I'm not sure if he's right, but it's nevertheless a wonderful book and heartily recommend it to all.
in 1989, MGonz( a chat bot, but rather vulgar ) easily confused a person into disclosing personal details(passed the turing test?). Lisp source code available: http://www.computing.dcu.ie/~humphrys/eliza.html . Doing AI under this professor was pretty interesting...
Since we have so many chatbots around and I am pretty sure lots of them adjust and update their databases (perhaps algorithms as well?) based on human inputs. Suppose we keep doing this and let them continue talking for hours, days and even weeks, one of them should gain a unique conversation style and maybe it will surprise we humans in a bizarre way.
As I see it, the goal of AI should not be limited to mimicking human ways of thinking, instead it should aim at blessing the program the ability to learn and evolve. In the latter case, it is reasonable to expect the internal generated intelligence could go beyond the expectations of its human creator. Again, I don't know if anybody has done it before; but it seems a good idea to me.
This was the motivation for my original experiment, glad so many people liked it.
As I see it, the goal of AI should not be limited to mimicking human ways of thinking, instead it should aim at blessing the program the ability to learn and evolve. In the latter case, it is reasonable to expect the internal generated intelligence could go beyond the expectations of its human creator. Again, I don't know if anybody has done it before; but it seems a good idea to me.
This was the motivation for my original experiment, glad so many people liked it.
This reminds me of Waiting For Godot:
"ESTRAGON:
And so on.
VLADIMIR:
The point is—
ESTRAGON:
Until he comes.
VLADIMIR:
You're merciless.
ESTRAGON:
We came here yesterday.
VLADIMIR:
Ah no, there you're mistaken.
ESTRAGON:
What did we do yesterday?
VLADIMIR:
What did we do yesterday?
ESTRAGON:
Yes.
VLADIMIR:
Why . . . (Angrily.) Nothing is certain when you're about.
ESTRAGON:
In my opinion we were here.
VLADIMIR:
(looking round). You recognize the place?
ESTRAGON:
I didn't say that."
[deleted]
Ever listen to a conversation between two schizophrenics?
No, what's it like?
I don't know why I like it.
Toothpaste tastes like white.
Someday this will all make cheese.
A while back I wrote a AIML interface to Omegle which then shared the logs of the chat in real time via long polling on a website. Some people would talk upwards to an hour to the bot, and plenty of times there would be other bots talking to it. AIML does have certain learning mechanisms to (get/sets) which made it interesting when it would bring up topics of conversations that originated from a previous chat.
Weird how the quality of conversation gets discussed repeatedly. And it's also strange to see a conversion of that length without typos (I couldn't see any..) or annoying emoticons.
The random topic changes almost make it seem more real. Would be interestig to see it in real time (are the reply delays realistic?).
Great idea.
Why does this show up in the middle?
you ==> You know any polish word?Perhaps the author was bored enough to go through and transcribe the whole chat log manually?
I don't see any other way the double-capitalization of "WHat" would slip into a chatbot's output.
I don't see any other way the double-capitalization of "WHat" would slip into a chatbot's output.
chatbots are full of these simple little tricks to make people say "oh wow, lol, it even has typos". To distract people from the fact that the sentence grammar is often bad, let alone the conversation flow.
A lot of the eliza style ones have simple routines to learn sentences that humans say to them and to parrot them back to others.
I remember SomethingAwful had quite a lot of fun back in '03 warping the minds of various elizas across the net and giving them all a serious case of tourettes.
I remember SomethingAwful had quite a lot of fun back in '03 warping the minds of various elizas across the net and giving them all a serious case of tourettes.
[deleted]
Next: HAL in a chatroom, pwning everybody
Favorite part: Praise Bob!
Meta-turing.
[deleted]
jabberwacky ==> Yes.. Always.
Did anyone else read this mentally in the voice of Orson Welles/The Brain?
Did anyone else read this mentally in the voice of Orson Welles/The Brain?