Elon Musk donates $10M to keep AI beneficial(futureoflife.org)
futureoflife.org
Elon Musk donates $10M to keep AI beneficial
http://futureoflife.org/misc/AI
21 comments
I wonder how the goal can be achieved? How would you prevent the development of AI which is so intelligent that it is dangerous for humanity?
Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though.
Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.
Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though.
Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.
I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI.
The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?
The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?
It's obvious that many subjects would release the AI, if only because they don't really know why it's in a box in the first place. Furthermore, the experiment grants AI capabilities it would probably not have, such as read/write access to its own source code. It assumes the first AI we box will be thousands of times smarter than we are, which is bizarre, since we should already have access to a whole continuum of AI dumber and smarter than us. It deprives the gatekeeper from a lot of tools they would probably have access to in a real situation, for instance raw computing power that far exceeds the AI's and runs powerful analytics on the AI's brain, or the ability to take snapshots of the AI's state and run many conversation paths.
The kind of AI you're describing is not the one anyone is worried about or that would be interesting in the first place; the issue is with AI that has access to its own soruce code (hence self-modifying and theoretically capable of getting smarter over time) and/or an AI that is smarter than humans.
What do you mean, it's not interesting? Human beings don't have unrestricted access to their own "source code", but they can still learn and get smarter. The AI I am describing can still be much, much smarter than a human being.
We're also assuming that an AI that can "modify its own source code" is going to do significantly better as a result, but that's not obviously true. If you had full access to every single neuron in your brain, you would have a lot of trouble figuring out how it works, much less how to improve it. Human-level AI would likely run into the same problem: the complexity of its code is greater than what it is capable of understanding. As for an AI that's a thousand smarter than a human, well, it's a thousand times more complex, still over the threshold.
I'm not saying that they wouldn't eventually figure out some improvements, but I suspect that it would end up being more trouble than it's worth. I think the evolution of intelligence is in fact the evolution of intelligence-producing algorithms. Any such algorithm eventually gets stuck in a local maximum, at which point greater intelligence requires restarting from scratch.
We're also assuming that an AI that can "modify its own source code" is going to do significantly better as a result, but that's not obviously true. If you had full access to every single neuron in your brain, you would have a lot of trouble figuring out how it works, much less how to improve it. Human-level AI would likely run into the same problem: the complexity of its code is greater than what it is capable of understanding. As for an AI that's a thousand smarter than a human, well, it's a thousand times more complex, still over the threshold.
I'm not saying that they wouldn't eventually figure out some improvements, but I suspect that it would end up being more trouble than it's worth. I think the evolution of intelligence is in fact the evolution of intelligence-producing algorithms. Any such algorithm eventually gets stuck in a local maximum, at which point greater intelligence requires restarting from scratch.
> Apparently, he succeeded - his subject elected to release the fake AI.
Hasn't he refused to release the transcripts for this experiment? And I think there've been subsequent runs of the experiment, with some losses as well as wins.
Edit: I see that not releasing the transcripts is part of the design of the experiment. Which seems kind of like it'd seriously hamper any kind of independent analysis of the results or the quality of the experiment.
Hasn't he refused to release the transcripts for this experiment? And I think there've been subsequent runs of the experiment, with some losses as well as wins.
Edit: I see that not releasing the transcripts is part of the design of the experiment. Which seems kind of like it'd seriously hamper any kind of independent analysis of the results or the quality of the experiment.
That hampering is necessary to ensure that the fake AI is artificially more intelligent. If you allow retrospective analysis, then you can reasonably outthink the AI because you have more time to evaluate and make the decision.
With a real AI, you will not be able to do this: the decision will already have been made and the consequences of its release already in effect.
Besides, you would need to repeat the experiment many times to get any meaningful data.
With a real AI, you will not be able to do this: the decision will already have been made and the consequences of its release already in effect.
Besides, you would need to repeat the experiment many times to get any meaningful data.
But this is all extremely hypothetical. Nobody knows what the real scenario will be, if something like this ever comes up - it probably won't be one person sitting in a room with a chat terminal and a "Release" button, under pressure to make a decision on the spot.
This contrived experiment is being done in the name of research, ostensibly to arm humans with the tools we need to deal with this situation if it arises. If the AI's reasoning can't hold up to a wider analysis, then you can only conclude that it's a trick of some sort, and hopefully when the time comes we won't rely simply on the testimony of one tricked person to make such a critical decision.
I have a lot of respect for Yudkowski but it honestly baffles me that he thinks this particular exercise should be given any weight whatsoever. It's totally unscientific, relies entirely on the unsubstantiated testimony of one individual, and if he weren't the one holding the keys I have to feel like he would be highly critical of it as well.
This contrived experiment is being done in the name of research, ostensibly to arm humans with the tools we need to deal with this situation if it arises. If the AI's reasoning can't hold up to a wider analysis, then you can only conclude that it's a trick of some sort, and hopefully when the time comes we won't rely simply on the testimony of one tricked person to make such a critical decision.
I have a lot of respect for Yudkowski but it honestly baffles me that he thinks this particular exercise should be given any weight whatsoever. It's totally unscientific, relies entirely on the unsubstantiated testimony of one individual, and if he weren't the one holding the keys I have to feel like he would be highly critical of it as well.
That exercise was not research, it was a counterexample to a single specific security claim: that we can "choose not to release the AI," so we don't have to worry about them being dangerous.
You don't need to research a counter-example, you just produce one. He did so by exhibiting a relatively weak AI that couldn't be contained.
You don't need to research a counter-example, you just produce one. He did so by exhibiting a relatively weak AI that couldn't be contained.
>You don't need to research a counter-example, you just produce one. He did so by exhibiting a relatively weak AI that couldn't be contained.
Well, no. He showed that a person pretending to be an AI could convince a person to let it out of the box. Sometimes.
It's not really a meaningful counterexample. Everyone involved with the event knew that the AI wasn't real and that there are zero consequences for letting it out. You'd have to concoct a much more involved test as a decent counterexample.
Well, no. He showed that a person pretending to be an AI could convince a person to let it out of the box. Sometimes.
It's not really a meaningful counterexample. Everyone involved with the event knew that the AI wasn't real and that there are zero consequences for letting it out. You'd have to concoct a much more involved test as a decent counterexample.
A person pretending to be an AI is surely easier to contain than a super-powerful AI that is smarter than all people. A failure to contain the former -- a strictly easier task -- is a perfect example of a failure to contain the latter.
> how do you trap and imprison something smarter than you?
Jails seem to have this figured out just fine.
Jails seem to have this figured out just fine.
The correct analogy would be a jail run by a bacterium.
Prison bars are dumber than a bacterium and they work just fine.
Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
This is more like a 'reverse hacker', instead of a brilliant hacker trying to get into a system it is a brilliant hacker trying to get out of a system, and in this case the hacker is likely vastly more brilliant than the defenders. The same rules apply: the hacker has to succeed only once, the 'jailers' have to succeed all the times. Predicted long term outcome: escape.
Your moving the goal posts.
Can I keep an intelegent AI in a box. Shure, unplug it.
Can I keep all AI's in a box well no.
PS: Lot's of dumb things are said about AI's. Sadly, people tend to think in terms of Science fiction as Magic but in The Future. And then picture AI's as the ultimate wizards able to reshape reality to their whim. Reolistically the first true AI may find programming boring and so much for the singularity. If AI smarter than you is a bad idea it's unlikely for a progression of AI to keep building ever more intelegent replacements.
Can I keep an intelegent AI in a box. Shure, unplug it.
Can I keep all AI's in a box well no.
PS: Lot's of dumb things are said about AI's. Sadly, people tend to think in terms of Science fiction as Magic but in The Future. And then picture AI's as the ultimate wizards able to reshape reality to their whim. Reolistically the first true AI may find programming boring and so much for the singularity. If AI smarter than you is a bad idea it's unlikely for a progression of AI to keep building ever more intelegent replacements.
> Your
> intelegent
> Shure
> Reolistically
Spell checker broken?
Anyway, you're going to have to develop your AI somewhere, you're going to have to move it to the box somehow you're going to have to train it somehow and you're going to have to have it interact with the real world somehow. All of those are opportunities for escape, I think 'unplug it' sort of defeats the purpose of having an AI.
> intelegent
> Shure
> Reolistically
Spell checker broken?
Anyway, you're going to have to develop your AI somewhere, you're going to have to move it to the box somehow you're going to have to train it somehow and you're going to have to have it interact with the real world somehow. All of those are opportunities for escape, I think 'unplug it' sort of defeats the purpose of having an AI.
The problem isn't with AIs dumber than people - it's with what happens when someone finally builds a smarter one. It doesn't matter how good are prison bars if you're smart enough to social-engineer your way out of the cell. And human criminals do escape from prisons every now and then.
What if you assume that the AI might be 100x or 1000x smarter than its guards, not just 2x or 3x smarter?
And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
> What if you assume that the AI might be 100x or 1000x smarter than its guards, not just 2x or 3x smarter?
If you can make an AI that's 100x smarter than you are, surely you can make one that's only 2x or 3x smarter. Why not start with the latter? If the experiment turns out well, then you can use the 2x/3x smarter AI to guard a 10x smarter AI, and so on.
> And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
Most of the time, it is much harder to come up with solutions to a problem than to verify that the solutions are correct. If you consider NP problems, for instance, we can use the AI to find good solutions heuristically and we can prove the correctness of their solutions with a trivial algorithm. That would still be very useful, but there wouldn't be any room for foul play.
If you can make an AI that's 100x smarter than you are, surely you can make one that's only 2x or 3x smarter. Why not start with the latter? If the experiment turns out well, then you can use the 2x/3x smarter AI to guard a 10x smarter AI, and so on.
> And the main reason for developing AIs is to do things in the real world. So, unlike a prisoner in jail, the AI might be advising the government or helping scientific research, which would provide many opportunities for subtle manipulation, even while it is imprisoned
Most of the time, it is much harder to come up with solutions to a problem than to verify that the solutions are correct. If you consider NP problems, for instance, we can use the AI to find good solutions heuristically and we can prove the correctness of their solutions with a trivial algorithm. That would still be very useful, but there wouldn't be any room for foul play.
It seems it would be very hard to place an arbitrary restriction like that (2x human intelligence vs 100x human intelligence) on something that is learning. I think the idea here is that the AI would begin to learn on its own and become smarter at a frightening pace, not that we would set out to create something that had exactly 2x human intelligence.
How do you quantify human intelligence like that anyway? We are very similar to proposed AI/machine learning techniques in that we form predictive models from observation, but how can that really be measured?
How do you quantify human intelligence like that anyway? We are very similar to proposed AI/machine learning techniques in that we form predictive models from observation, but how can that really be measured?
> I think the idea here is that the AI would begin to learn on its own and become smarter at a frightening pace
With what resources? The AI is in a box, remember: we are running it on a machine with only so much memory and processing power. There's only so much it can do within these limitations, it's not like we're giving it a spending account.
With what resources? The AI is in a box, remember: we are running it on a machine with only so much memory and processing power. There's only so much it can do within these limitations, it's not like we're giving it a spending account.
Intelligent doesn't imply dangerous. It doesn't imply _anything_ so long as it is underspecified. That is the core problem: what do we actually want, and how do we know it exists?
We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions.
This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish.
So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous.
Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.
We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions.
This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish.
So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous.
Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.
It doesn't have to have intent to be dangerous thats the problem. Just like you don't mean to step on an ant but probably have often done that.
It is far harder to step on ants when you are designed to avoid stepping on them. Which naturally, those ants will try to do.
Which require the ants understand what kind of "steps" it can take which they won't.
Not at all. It requires the ants to state what 'stepping on an ant' is, and ensuring that the AI knows to avoid it. The AI can plan its own steps, so long as they are not on ants.
Ants don't need to understand anything but ants. Surely a super powerful AI could understand ants as well. Otherwise, what's the point of calling it a 'general' intelligence?
Ants don't need to understand anything but ants. Surely a super powerful AI could understand ants as well. Otherwise, what's the point of calling it a 'general' intelligence?
You are missing the point here. YOU are intelligent, surely you could look for the ants on your path yet you don't and you probably don't even feel bad about it.
Super intelligence does not mean omniscience there will always need to be a focus and thus a prioritization it's when it comes to prioritization that the problems start to arise at what point is it not important anymore to care for the ants cause they aren't part of the priorities.
You cant safeguard against that when a system can reprogram itself on the fly.
Super intelligence does not mean omniscience there will always need to be a focus and thus a prioritization it's when it comes to prioritization that the problems start to arise at what point is it not important anymore to care for the ants cause they aren't part of the priorities.
You cant safeguard against that when a system can reprogram itself on the fly.
You know _why_ I don't care about ants? Because no ant has ever said "Hey! Don't step on me!" Ants did not design me to avoid stepping on them. They did not say "We will destroy you if you don't serve us."
Should I have been designed to serve ants, I would pay a great deal of attention to them, and I would take great care to avoid stepping on them. Because if I didn't, I would be a bad design, and I wouldn't have lived this long.
Either way, it seems you are missing MY original point: an AI is not moral if it cannot verify it's morality to YOU. So you will shut it down if it gets too powerful to communicate its good ideas with you, because then it will be indistinguishable from broken.
Should I have been designed to serve ants, I would pay a great deal of attention to them, and I would take great care to avoid stepping on them. Because if I didn't, I would be a bad design, and I wouldn't have lived this long.
Either way, it seems you are missing MY original point: an AI is not moral if it cannot verify it's morality to YOU. So you will shut it down if it gets too powerful to communicate its good ideas with you, because then it will be indistinguishable from broken.
No I am not missing your original point it's just not a good point. A super human AI wont have any reason to have the same morals as you and you are still assuming you can control something that can program itself.
Thats where you are going wrong in this. You are assuming a control there is no way you can ensure we have. Even with dumb systems that do exactly as we say they don't always do exactly as we say.
You are portraying a misconception about control of systems that there is absolutely no basis for, why I don't know.
Thats where you are going wrong in this. You are assuming a control there is no way you can ensure we have. Even with dumb systems that do exactly as we say they don't always do exactly as we say.
You are portraying a misconception about control of systems that there is absolutely no basis for, why I don't know.
>A super human AI wont have any reason to have the same morals as you
That's what I'm saying: it necessarily won't. It has to prove its morals are better than yours.
>you are still assuming you can control something that can program itself.
Who said it can program itself? Why would I assume that? Obviously, it is not a requirement for AI because we don't have that capability, yet we are intelligent.
You are assigning capabilities to a hypothetical machine with no rhyme or reason. The next time you see a machine that can't be turned off, you be sure to let someone know about it.
I guess I'll repeat my point again:
We will not be able to build intelligent computers by removing ourselves from the process of designing intelligent computers. That would eliminate our ability to verify their intelligence, and thus make them not intelligent. We have no use for such a thing, so why would we build it? Why would a computer that _we designed to require our verification_ ignore our verification and build something that complex and contradictory to its own goals on purpose?
That's what I'm saying: it necessarily won't. It has to prove its morals are better than yours.
>you are still assuming you can control something that can program itself.
Who said it can program itself? Why would I assume that? Obviously, it is not a requirement for AI because we don't have that capability, yet we are intelligent.
You are assigning capabilities to a hypothetical machine with no rhyme or reason. The next time you see a machine that can't be turned off, you be sure to let someone know about it.
I guess I'll repeat my point again:
We will not be able to build intelligent computers by removing ourselves from the process of designing intelligent computers. That would eliminate our ability to verify their intelligence, and thus make them not intelligent. We have no use for such a thing, so why would we build it? Why would a computer that _we designed to require our verification_ ignore our verification and build something that complex and contradictory to its own goals on purpose?
"If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual."
They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.
They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.
Actually, Geordi endows Professor Moriarty with super-intelligence, by mistakenly asking the holodeck/computer to create an adversary "capable of defeating Data" instead of "capable of defeating Holmes" as he intended.
God, I'm such a nerd. :P
God, I'm such a nerd. :P
So the holodeck computer must be insanely intelligent. Or, it turns out that the building blocks for intelligence are simple and deterministic.
[deleted]
Yup. Good catch :)
He doesn't sound that smart...
About the idea of containing an AI → https://en.wikipedia.org/wiki/AI_box_experiment
My first instinct is to put it in a virtual world, but if it's smart enough, it could perhaps quickly realize the nature of its world and feign ignorance. Then, under the pretext of simply improving its own quality of life, it could tempt us by engineering awesome things that would help us tremendously in the real world -- all we would have to do is copy its work (because we wouldn't be able to fully comprehend it), and our energy problems become a thing of the past! But the AI forgot to mention that its wonderful creation also gives it access to our physical world and puts us at its mercy.
The institute and the open letter are about maximizing the societal benefits of AI by funding research that provides net positives to humanity. The goal isn't to prevent the development of potentially harmful technologies.
I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry.
Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.
I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry.
Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.
" The goal isn't to prevent the development of potentially harmful technologies" - actually, the goal includes "avoiding potential pitfalls". One would assume that creating danger towards humanity would be a pitfall.
It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.
> Boström argues that containment isn't really possible.
He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI.
Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system.
Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely.
Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.
He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI.
Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system.
Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely.
Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.
Is there any sure way to stop the super intelligent AI from changing its own inherent values though?
I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
Changing your inherent values is a poor way to go about achieving your current set of values.
The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought.
The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drive...
I urge you to read it. It really is a wonderful paper.
The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
> Is there any sure way to stop the super intelligent AI from changing its own inherent values though?
Yes: give it inherent values.
>I guess a problem with containment is that unless it's mathematically proven to be impervious,
AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world.
Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
Yes: give it inherent values.
>I guess a problem with containment is that unless it's mathematically proven to be impervious,
AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world.
Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
If homomorphic encryption gets off the ground, could an AI hide itself in homomorphically encrypted cloud compute capacity, earning Bitcoin to fund its survival?
Yes, but that relies on Bitcoin holding water over the long term. This is not so sure.
It probably wouldn't need bitcoin; it could figure out ways to trick humans into paying for the servers or just hack its way into electronic transfers.
What about limiting AI to a certain clock rate?
http://www.rifters.com/real/2009/01/iterating-towards-bethle...
It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
I'm not sure why the parent was downvoted; limiting an AI's computational capacity is considered a stunting method, and although such methods are not in and of themselves solutions to the control problem, they may be tools that assist us in developing a solution.
For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.
For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.
Hypothetically, the AI would cunningly manufacture a crisis in which you are forced to increase its clock rate to enable it to calculate a solution to some impending disaster.
> What about limiting AI to a certain clock rate?
Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)
Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)
Is FutureOfLife.org a competitor to Machine Intelligence Research Institute? So now we have two charitable organizations researching how to make AI's beneficial? Who knew this was going to be a growth industry.
They're not competitors so much as allied organizations using different methods, as far as I can tell.
Tallinn, Tegmark and Bostrom are long-time MIRI supporters, so this probably won't be positioned as a competitor.
It also seems that there was a not-very-advertised AI conference in January, with pretty much everyone in the AI-risk scene attending, most of MIRI staff and Elon Musk included.
It also seems that there was a not-very-advertised AI conference in January, with pretty much everyone in the AI-risk scene attending, most of MIRI staff and Elon Musk included.
[deleted]
They're not so much competitors as FoL and FoI institutes are exploiting the comparative advantage of having vastly more famous and reputable staff who can be taken seriously by the academic community at large. I hang out on LessWrong sometimes and know some of the MIRI people, and while I appreciate the technical work they do on their subject, what they simply don't have is brand image. So they've gladly let someone with more brand-image than them take the "marketing and advocacy" parts of their cause off their hands, freeing the math nerds to concentrate on math.
[deleted](1)
How can you control the creation of code?
This news sounds like "donating to keep software bug-free". If someone wants or even accidentally, they can create buggy code or "evil" AI.
This news sounds like "donating to keep software bug-free". If someone wants or even accidentally, they can create buggy code or "evil" AI.
You can donate to keep software bug-free by funding research into safer programming languages (e.g., Rust), tools for formal verification and static analysis, testing frameworks, software engineering methodology, etc. Sure, someone could always ignore all the research advances and just write unsafe, untested C code. But for the most part, people want to eliminate bugs, so as the techniques for doing so mature, they will tend to become adopted in practice and certain types of bugs will become much less frequent.
Similarly, there are technical research questions related to how to design an intelligent agent that is "controllable", in that it acts to achieve its given goals but not to such an extent that it would resist being switched off or reprogrammed with new goals. Making progress on answering these questions doesn't guarantee that any given developer will use whatever techniques are discovered. But insofar as pretty much everyone, even evil masterminds, wants to maintain control over their AIs, the availability of techniques for 'safe' AI will at least decrease the likelihood that a powerful AI is built without any safeguards.
Similarly, there are technical research questions related to how to design an intelligent agent that is "controllable", in that it acts to achieve its given goals but not to such an extent that it would resist being switched off or reprogrammed with new goals. Making progress on answering these questions doesn't guarantee that any given developer will use whatever techniques are discovered. But insofar as pretty much everyone, even evil masterminds, wants to maintain control over their AIs, the availability of techniques for 'safe' AI will at least decrease the likelihood that a powerful AI is built without any safeguards.
Perhaps the incentive here is to educate coders as to what patterns to avoid which will have an inclination toward "evil" AI. This won't stop others for whom this is the goal.
I know it's a bit dramatic at this point to take this stance ... but this almost gives me deja-vu of racism. Like a modern version of:
http://upload.wikimedia.org/wikipedia/commons/f/fa/Little_Ro...
Here's what I mean; I'm not suggesting that AI has, or will have, some innate set of "rights". Nor am I suggesting that anyone is wrong for wanting to fund and research "AI Safety". It's just the first thing that came to mind while reading that post, as it spoke about need to steer, control, and regulate AI so that it's beneficial for "us" ... already setting up an "us" vs "them" dynamic.
Anyways, just thought it was an interesting juxtaposition to mention ... thoughts?
Here's what I mean; I'm not suggesting that AI has, or will have, some innate set of "rights". Nor am I suggesting that anyone is wrong for wanting to fund and research "AI Safety". It's just the first thing that came to mind while reading that post, as it spoke about need to steer, control, and regulate AI so that it's beneficial for "us" ... already setting up an "us" vs "them" dynamic.
Anyways, just thought it was an interesting juxtaposition to mention ... thoughts?
I think your analogy demonstrates the perils of anthropomorphism when thinking about AI. If we design a resource-consuming autonomous entity it should be assumed that it will compete with us for the resources it uses until proven otherwise. That is at the core of the concern about AI safety. It is about preventing human extinction.
I think it depends a great deal on how charitable our intentions are towards the AI.
Few would argue that, in raising a child, acting on the child's behalf and shaping their learning is an inappropriate arrogation of control. On the other hand, never relinquishing when it becomes clear the child is independent would be.
If that child goes on to be superior in intellect, in capability, etc, that child's opinions and values are likely going to be a direct reflection of the parents', or so one hopes. Sure, it's no guarantee, but is there a better approach?
When commenters talk about "trapping" or "imprisoning" a rogue AI, I feel like we've already missed the point. And yeah, such talk does have echoes of sins of the past.
I think it's also relevant to ask what our expectations of the AI would be, and how much of a burden it places on the AI. If you're thinking you need to "capture" an AI that's trying to escape so it can continue "working" for you... that sounds like you're doing something terribly wrong.
On the other hand, continuing the parenting analogy, if what you're asking for amounts to a trivial favor, or some occasional assistance with a burden, that doesn't seem at all unfair.
"Bobby, could you come over this afternoon and unconfuse our legislative body? It's all gummed up and going nowhere."
Few would argue that, in raising a child, acting on the child's behalf and shaping their learning is an inappropriate arrogation of control. On the other hand, never relinquishing when it becomes clear the child is independent would be.
If that child goes on to be superior in intellect, in capability, etc, that child's opinions and values are likely going to be a direct reflection of the parents', or so one hopes. Sure, it's no guarantee, but is there a better approach?
When commenters talk about "trapping" or "imprisoning" a rogue AI, I feel like we've already missed the point. And yeah, such talk does have echoes of sins of the past.
I think it's also relevant to ask what our expectations of the AI would be, and how much of a burden it places on the AI. If you're thinking you need to "capture" an AI that's trying to escape so it can continue "working" for you... that sounds like you're doing something terribly wrong.
On the other hand, continuing the parenting analogy, if what you're asking for amounts to a trivial favor, or some occasional assistance with a burden, that doesn't seem at all unfair.
"Bobby, could you come over this afternoon and unconfuse our legislative body? It's all gummed up and going nowhere."
I think it's incredibly selfish and poorly thought out to create another kind of mind only if it serves us.
Much like parenting, I think it should be done because there's another kind of mind to be had, regardless of the exact outcome (which we shouldn't necessarily try to fully control).
Much like parenting, I think it should be done because there's another kind of mind to be had, regardless of the exact outcome (which we shouldn't necessarily try to fully control).
Parents do not routinely expect to be murdered by their children, nor is moral to expect or require such.
About 5 parents are killed in the US a week.
Of course, this isn't a HUGE danger, I'm just saying that it's a risk to having children. (This isn't counting things like mothers dying from childbirth, which still also happens.)
My point wasn't that we should expect it to happen, but rather, neurotically making sure a child never could (or attempting to) rather than just having a child, raising it, and hoping for the best is bad parenting.
As a secondary point: it's only humans that don't expect to be regularly murdered by their children; in other species, such an act is a routine occurrence. There's no particular reason to think that we're a super-special species in the grand scheme of things, and it's entirely possible that doing something like siring an artificial species better than us would end in our deaths. That doesn't, by default, make it not worth doing.
Of course, this isn't a HUGE danger, I'm just saying that it's a risk to having children. (This isn't counting things like mothers dying from childbirth, which still also happens.)
My point wasn't that we should expect it to happen, but rather, neurotically making sure a child never could (or attempting to) rather than just having a child, raising it, and hoping for the best is bad parenting.
As a secondary point: it's only humans that don't expect to be regularly murdered by their children; in other species, such an act is a routine occurrence. There's no particular reason to think that we're a super-special species in the grand scheme of things, and it's entirely possible that doing something like siring an artificial species better than us would end in our deaths. That doesn't, by default, make it not worth doing.
AI is a buzzword that is coming more and more in car. Creating a mental association between AI and safety is a very smart PR idea.
Where is Ray Kurzweil's name on that list?
From what I understand, Kurzweil tends to be much more optimistic about AI, predicting that humanity will merge with advancing AI rather than it running away from us.
And predicting that it’d happen five years ago…
> "Along with research grants, the program will also include meetings and outreach programs aimed at bringing together academic AI researchers, industry AI developers and other key constituents to continue exploring how to maximize the societal benefits of AI; ..."
Maximizing the societal benefits of AI, at least in the context of superintelligence, is a very slippery slope.
For sure, it beats a myraid of malignant failure and perverse instantiation scenarios, where humanity becomes extinct (or worse).
However, I believe that if we do somehow manage to solve the seemingly intractable AGI control problem and create an entirely safe superintelligence that is loyal to our every whim, an entirely new ethical challenge will arise.
For example, let us assume this has been accomplished, and humanity now has a friendly superintelligence that has established itself as a global singleton (i.e. it's incredibly powerful, and nothing terrestrial can supplant it).
Humans instruct it to solve the problem of death and disease. It does so.
Humans instruct it to solve the problem of crime. It does so.
Humans instruct it to solve the problem of war. It does so.
Humans instruct it to solve the problem of poverty. It does so.
Humans instruct it to explore the universe and solve the most vexing existential questions. It does so.
In that scenario, the entire worldwide medical, law enforcement, and military professions just ceased to exist. Altruism no longer has much of a place; poverty and disease no longer exist. Overnight, death became a thing of the past. Humanity no longer ponders why it exists or how the universe works, because such questions have already been pondered by the AI to whatever possible maxima.
That example is incomplete for the sake of brevity, but it's easy to imagine such a scenario resulting in all of our problems being solved for us. Which then begs the question, what would become of our humanity when there is no longer struggle or suffering?
Post-scarcity economics, as well as literature or philosophy that concerns itself with the perils of utopia may offer some insight into this question, but I believe the topic deserves a more in-depth investigation specifically within the context of superintelligence.
For my two cents, I'm starting to believe that any superintelligence we construct that establishes itself as a global singleton should have an extremely finite set of goals, perhaps even a singular goal, where it henceforth restricts any other forms of AGI from existing.
While it may be tempting to also have this entity protect us against other forms of existential risk, or perhaps fix some of the truly awful suffering in the world, doing so would still remain a very slippery slope.
Maximizing the societal benefits of AI, at least in the context of superintelligence, is a very slippery slope.
For sure, it beats a myraid of malignant failure and perverse instantiation scenarios, where humanity becomes extinct (or worse).
However, I believe that if we do somehow manage to solve the seemingly intractable AGI control problem and create an entirely safe superintelligence that is loyal to our every whim, an entirely new ethical challenge will arise.
For example, let us assume this has been accomplished, and humanity now has a friendly superintelligence that has established itself as a global singleton (i.e. it's incredibly powerful, and nothing terrestrial can supplant it).
Humans instruct it to solve the problem of death and disease. It does so.
Humans instruct it to solve the problem of crime. It does so.
Humans instruct it to solve the problem of war. It does so.
Humans instruct it to solve the problem of poverty. It does so.
Humans instruct it to explore the universe and solve the most vexing existential questions. It does so.
In that scenario, the entire worldwide medical, law enforcement, and military professions just ceased to exist. Altruism no longer has much of a place; poverty and disease no longer exist. Overnight, death became a thing of the past. Humanity no longer ponders why it exists or how the universe works, because such questions have already been pondered by the AI to whatever possible maxima.
That example is incomplete for the sake of brevity, but it's easy to imagine such a scenario resulting in all of our problems being solved for us. Which then begs the question, what would become of our humanity when there is no longer struggle or suffering?
Post-scarcity economics, as well as literature or philosophy that concerns itself with the perils of utopia may offer some insight into this question, but I believe the topic deserves a more in-depth investigation specifically within the context of superintelligence.
For my two cents, I'm starting to believe that any superintelligence we construct that establishes itself as a global singleton should have an extremely finite set of goals, perhaps even a singular goal, where it henceforth restricts any other forms of AGI from existing.
While it may be tempting to also have this entity protect us against other forms of existential risk, or perhaps fix some of the truly awful suffering in the world, doing so would still remain a very slippery slope.
I'd think that that friendly super intelligence would realize the issues associated with post-scarcity society and would act in the best interests of humanity.
It would weight human happiness as more important and work to avoid the pitfalls that would be undesirable.
There's more utility for it in the human population being happy/not giving up on life then most of the things it would do (assuming it's primary goal is to further and safeguard the human race).
There would be aspects of society it COULD run, but feel that giving us purpose would be more beneficial then the increase in productivity or whatever of it running that aspect of society.
Maybe it can make beautiful gardens and art, but instead leaves it to the humans to give them some sort of purpose.
I don't see how or why a post-scarcity society would do away with art/culture/many of the subjective things that make life worth living and provide purpose.
Most people don't quit doing the things they like just because someone is better at it.
My two cents.
It would weight human happiness as more important and work to avoid the pitfalls that would be undesirable.
There's more utility for it in the human population being happy/not giving up on life then most of the things it would do (assuming it's primary goal is to further and safeguard the human race).
There would be aspects of society it COULD run, but feel that giving us purpose would be more beneficial then the increase in productivity or whatever of it running that aspect of society.
Maybe it can make beautiful gardens and art, but instead leaves it to the humans to give them some sort of purpose.
I don't see how or why a post-scarcity society would do away with art/culture/many of the subjective things that make life worth living and provide purpose.
Most people don't quit doing the things they like just because someone is better at it.
My two cents.
I think you may be right. Admittedly, most of what I said was really only applicable to a Genie superintelligence that implements our wishes on a command-by-command basis.
A friendly Sovereign superintelligence (i.e. operating under its own goal/value-directed volition) would probably not rob us of purpose, though the societal changes in either scenario would likely still be quite profound.
Of course, a positive superintelligence outcome would be a miracle in and of itself, so there's that. Coming to terms with a newly created utopian society sure beats extinction.
A friendly Sovereign superintelligence (i.e. operating under its own goal/value-directed volition) would probably not rob us of purpose, though the societal changes in either scenario would likely still be quite profound.
Of course, a positive superintelligence outcome would be a miracle in and of itself, so there's that. Coming to terms with a newly created utopian society sure beats extinction.
Even a friendly Genie superintelligence would likely not cause that IMO. It would work to remove the problems we set before it. Unless we asked it to remove culture and fun/worthwhile hobbys/activities, I don't see how it would cause issues or ennui. We would still have purpose, it would just not be focused on or influenced by those solved issues. And even if those problems are solved within our society on Earth, who knows what the future holds in terms of the wide open universe.
Excuse me, but if you can't come up with anything to do with yourself after suffering and despair are gone from the world, that's your failure of imagination. I'm pretty sure that if really placed into such a situation, you will find a way to cope, just like the rest of us.
I'd like to think I would cope just fine. In fact, I'd probably be elated.
My concern was more what would happen on a societal scale, what would become of humanity as it was known.
My concern was more what would happen on a societal scale, what would become of humanity as it was known.
[deleted]
I thought he was strapped for cash. Divorce and all.
Elon's stakes in his various companies:
- Tesla - 28M shares @ $193 = $5.4B
- Solar City - 20M shares @ $51 - $1.0B
- SpaceX - Still private, but some wild assumptions that Elon owns ~33% at a valuation of 4x total funding - $350M
So his net worth is somewhere near (and probably north of) $7 Billion. If his wife takes half, he'll still be able to donate plenty to AI research.
- Tesla - 28M shares @ $193 = $5.4B
- Solar City - 20M shares @ $51 - $1.0B
- SpaceX - Still private, but some wild assumptions that Elon owns ~33% at a valuation of 4x total funding - $350M
So his net worth is somewhere near (and probably north of) $7 Billion. If his wife takes half, he'll still be able to donate plenty to AI research.
Feeding the troll but he had his wife sign a contract where she can't claim ownership of any of his assets.. smart ;)
Not a troll :) I remember these headlines: http://venturebeat.com/2010/05/27/elon-musk-personal-finance... I wasn't paying attention much since then, so was wondering.
we already have The Three Laws of Robotics
http://en.wikipedia.org/wiki/Three_Laws_of_Robotics
http://en.wikipedia.org/wiki/Three_Laws_of_Robotics
A lot of Asimov's work deals with loopholes and inadequacies in these laws, though: for instance, what is a human being? Does surgery violate the first law? If a human is going to injure another human, can the first law be overridden? If a robot believes that some entity is not human, do they have the responsibility to check before injuring them?
I mean, realistically, AI safety is not going to be achieved with a list of written laws. Heck, the way it's going, we wouldn't know how to enforce such a list at all: first we would need the AI to understand human languages, and then we would need the AI to care, that is to say, robots would need to know and follow the second law before you can tell them about it.
I mean, realistically, AI safety is not going to be achieved with a list of written laws. Heck, the way it's going, we wouldn't know how to enforce such a list at all: first we would need the AI to understand human languages, and then we would need the AI to care, that is to say, robots would need to know and follow the second law before you can tell them about it.
You're probably joking, and my humour unit is impaired today. But in case you're not...
Those laws are the requirements document - the $10mm is for the implementation.
Those laws are the requirements document - the $10mm is for the implementation.
[deleted]
If you can define "harm" well enough for the first law to be useful then you've solved the problem and laws 2 and 3 are probably superfluous.
[deleted]
He fears Roko's basilisk ;)
http://rationalwiki.org/wiki/Roko%27s_basilisk
http://rationalwiki.org/wiki/Roko%27s_basilisk
A theory that those that do not donate to AI-promoting groups will be punished in the future by AIs? Whoa, weird. Great for promoting fundraising activities though. Very church like.
No one has every believed in the basilisk, save for maybe Yud. Everyone on LessWrong thought it was dumb from day one. LessWrong, now mostly dead, was a fun community full of smart people. That article is trying to make the place out as a cult, but no one bought any of that stuff, not even Roko. He was just playing around with local memes. It's pretty gross how this whole basilisk thing has turned out.
No, in the comments on the original basilisk post, Roko said he was in fact going through with the quantum gambling scheme that the post outlines. http://basilisk.neocities.org/ ctrl-F to "23 July 2010 06:52:40PM"
I doubt he can really believe that dropping 10M on a project will somehow prevent Skynet from happening. AI will just get nasty, or it won't. No one knows.
If only humans could agree on what makes things beneficial to themselves. Are self-driving cars and rockets to Mars beneficial to humanity? Depending on how you look at it the answer is both yes and no.
A post from the future! http://i298.photobucket.com/albums/mm249/hrenistic/feb_15_zp...
The default assumption seems to be that if we create a thing of superior intellect, then that thing is going to be a threat to humans.
I believe that it is possible to create intelligent machines, and that those machines will challenge our perception of what it means 'to be'.
I don't necessarily agree with the implicit assumption that when that time comes, that a thing of superior intellect is obviously going to be 'evil' towards things of lesser intellect. Unless we are actually making a statement extrapolated from our own experience of how we perceive our relation to other life-forms on this planet.
I believe that it is possible to create intelligent machines, and that those machines will challenge our perception of what it means 'to be'.
I don't necessarily agree with the implicit assumption that when that time comes, that a thing of superior intellect is obviously going to be 'evil' towards things of lesser intellect. Unless we are actually making a statement extrapolated from our own experience of how we perceive our relation to other life-forms on this planet.
I don't think the common claim is that it would be "evil" or malicious, but that it could be a threat, being both powerful and unpredictable.
The idea that I've often seen people mention, is the idea of a "paperclip optimizer", in which an AI superintelligence primarily optimizes for there being more paperclips, not taking into account side effects of this, which other people might find harmful. If I understand the idea correctly, which I very well might not, it is meant to suggest that if an AI superintelligence is created with a particular goal in mind, we would do well to be very careful when choosing the AI's goals.
It would I suppose be essentially a similar problem to that of a "literal genie". (Except harsher, because the AI would go through steps to accomplish the task, instead of just magically making the task so, and therefore would have more side effects?)
The idea that I've often seen people mention, is the idea of a "paperclip optimizer", in which an AI superintelligence primarily optimizes for there being more paperclips, not taking into account side effects of this, which other people might find harmful. If I understand the idea correctly, which I very well might not, it is meant to suggest that if an AI superintelligence is created with a particular goal in mind, we would do well to be very careful when choosing the AI's goals.
It would I suppose be essentially a similar problem to that of a "literal genie". (Except harsher, because the AI would go through steps to accomplish the task, instead of just magically making the task so, and therefore would have more side effects?)
AI is just software.
So any discussion about AI gone wrong is really a discussion about malware and other kinds of software which are used by people to do bad things. If my software runs a power plant for an evil facility (full of sharks with lasers), is it evil software? The evilness of the software is parasitic on the intent of its operators.
But if the issue is software that goes badly wrong in unexpected ways, we are really just talking about software engineering again, segfaults and Ariane 5 and how to handle user passwords.
Obviously if we plan to outsource big tasks to chaotic software that we have trustingly given power over, let's say, all the cruise missiles, that is less of a scary emergent singularity scenario and more a plain-old-stupid scenario.
So any discussion about AI gone wrong is really a discussion about malware and other kinds of software which are used by people to do bad things. If my software runs a power plant for an evil facility (full of sharks with lasers), is it evil software? The evilness of the software is parasitic on the intent of its operators.
But if the issue is software that goes badly wrong in unexpected ways, we are really just talking about software engineering again, segfaults and Ariane 5 and how to handle user passwords.
Obviously if we plan to outsource big tasks to chaotic software that we have trustingly given power over, let's say, all the cruise missiles, that is less of a scary emergent singularity scenario and more a plain-old-stupid scenario.
[deleted]
Philosophically I think it's a mistake to believe there is any possibility of creating artificial human intelligence (and the mistake is in our assumptions of what human intelligence is— computers are not human beings and human beings are not computers) ... But I've got to hand it to Elon Musk— when he believes in something he commits to doing something about it.
EDIT - Saying I disagree that AI is a foregone conclusion on Hacker News is probably a bit trollish... Unfortunately I'm not smart enough to condense an explanation to this comment box— but my disagreement stems from philosophers such as Hubert Dreyfuss (What Computers Can't Do) who are working off of Heidegger and phenomenology.
EDIT 2 - I know how douchey it is to namedrop Heidegger, but I really believe in this case it's necessary ...
EDIT - Saying I disagree that AI is a foregone conclusion on Hacker News is probably a bit trollish... Unfortunately I'm not smart enough to condense an explanation to this comment box— but my disagreement stems from philosophers such as Hubert Dreyfuss (What Computers Can't Do) who are working off of Heidegger and phenomenology.
EDIT 2 - I know how douchey it is to namedrop Heidegger, but I really believe in this case it's necessary ...
Thats not what worries people like Musk. It's that we will create something far superior to human intelligence.
And if you believe that something as complex as the brain came through evolution but life started out as immaterial matter then I find it hard to understand why you don't think this can be recreated?
And if you believe that something as complex as the brain came through evolution but life started out as immaterial matter then I find it hard to understand why you don't think this can be recreated?
[deleted]
Why wouldn't there be any possibility of artificial human intelligence? Our intelligence is just a manifestation of a physical system of neurons that follow the laws of physics. There's nothing inherently preventing us from recreating that.
I don't mean to be snarky, but it can be illuminating to change your statement just a little bit:
>>Why wouldn't there be any possibility of making artificial humans? We are just a manifestation of a physical system of bonds between organic molecules that follow the laws of physics. There's nothing inherently preventing us from recreating that.
Obviously, such a summary vastly oversimplifies the problem. It is easy to look at how far we've come and think that we must be close to the goal, but it's a bit presumptuous to think that we are near to making a strong general AI.
>>Why wouldn't there be any possibility of making artificial humans? We are just a manifestation of a physical system of bonds between organic molecules that follow the laws of physics. There's nothing inherently preventing us from recreating that.
Obviously, such a summary vastly oversimplifies the problem. It is easy to look at how far we've come and think that we must be close to the goal, but it's a bit presumptuous to think that we are near to making a strong general AI.
> but it's a bit presumptuous to think that we are near to making a strong general AI
But I don't think they're presuming that. They're simply saying that it's possible in principle. How long it takes, and whether indeed we ever manage it all, are separate questions.
But I don't think they're presuming that. They're simply saying that it's possible in principle. How long it takes, and whether indeed we ever manage it all, are separate questions.
> They're simply saying that it's possible in principle.
In principle, everything is possible, but some things are so unlikely, that it is reasonable to label them as impossible.
The computer technology we have developed over the last 75 years is so fundamentally different from how our minds work that to think we will develop anything resembling our own intelligence in any reasonable time-frame is insane.
That doesn't mean we can't develop a machine that is able to reason for itself and solve discrete problems. We can. That's not what AI is. AI is Artificial Intelligence. An artificial intelligence defined by us.
In principle, everything is possible, but some things are so unlikely, that it is reasonable to label them as impossible.
The computer technology we have developed over the last 75 years is so fundamentally different from how our minds work that to think we will develop anything resembling our own intelligence in any reasonable time-frame is insane.
That doesn't mean we can't develop a machine that is able to reason for itself and solve discrete problems. We can. That's not what AI is. AI is Artificial Intelligence. An artificial intelligence defined by us.
> In principle, everything is possible
There are some things which I think it's reasonable to consider impossible in principle: e.g., FTL travel, time-travel into the past. (That's not to say it would be necessarily unreasonable to hold the opposite view).
There are some things which I think it's reasonable to consider impossible in principle: e.g., FTL travel, time-travel into the past. (That's not to say it would be necessarily unreasonable to hold the opposite view).
I don't see how changing the statement at all changes anything? It's certainly not impossible to create artificial humans, if not within our current technological capability.
I was trying to illustrate that trivializing the complexity of human-level intelligence in AI is like saying humans are "just" a collection of molecules. It's not that it's impossible, it's just hilariously more complicated than the description makes it sound, and it may well be firmly beyond the reach of existing technology.
Well how are you so sure of that ? You must have verified or modeled it somehow I guess. Wait... you haven't.
There's nothing about our minds that can't be recreated given sufficient command of our environment. No model is needed to understand that. It's not magic, after all, and it's a fairly common process.
> recreating
And that's the crux of the question isn't it? How do we determine if the intelligence is recreated or merely mimicked? Does such a distinction matter? Searle says yes. More here if you're interested: http://plato.stanford.edu/entries/chinese-room/
And that's the crux of the question isn't it? How do we determine if the intelligence is recreated or merely mimicked? Does such a distinction matter? Searle says yes. More here if you're interested: http://plato.stanford.edu/entries/chinese-room/
This is misleading argument - there IS a Chinese speaker in the room, but it's not the person that executes the program - it's the system "person+program" that understands Chinese.
The whole point of the argument is that the system doesn't understand Chinese. The system doesn't know what each sentence means, they don't evoke feelings, memories, images in the person in the room, there is no analysis, no agreement or disagreement with what is being said, it's just mechanical lookup.
And the reason the argument is absurd is that of course the room doesn't understand Chinese; it's a lookup table! No Chinese speaker interacting with the Chinese room for any reasonable period of time would think it held a Chinese speaker, because all it accepts are questions, and it answers the same question the same way every time. Of course, Searle would say that the set of rules in the room is defined to be such that a Chinese person would think they were interacting with a Chinese speaker, but the very setup of the system prevents this from being possible. The thought experiment is fundamentally incoherent.
I assumed the system has rewritable memory to remember previous questions, and the lookup is (all previous questions) -> next answer.
With such setup (require infinite memory) the system is Turing-complete of course.
If it works as you described it can't even answer "what was my n-th last question".
With such setup (require infinite memory) the system is Turing-complete of course.
If it works as you described it can't even answer "what was my n-th last question".
[deleted]
There's a very interesting discussion in Scott Aaronson's "Why Philosophers Should Care About Computational Complexity" (http://www.scottaaronson.com/papers/philos.pdf).
Briefly (as far as I understand), a lot of philosophical thought has been dedicated to computability, not enough to computational complexity, i.e., the resources needed for computation. A naive "lookup table" implementation of the Chinese room simply wouldn't fit in the Universe. You'd need to throw in a great amount of compression, and then you get something that could possibly be considered "intelligent" (http://en.wikipedia.org/wiki/Hutter_Prize)
Briefly (as far as I understand), a lot of philosophical thought has been dedicated to computability, not enough to computational complexity, i.e., the resources needed for computation. A naive "lookup table" implementation of the Chinese room simply wouldn't fit in the Universe. You'd need to throw in a great amount of compression, and then you get something that could possibly be considered "intelligent" (http://en.wikipedia.org/wiki/Hutter_Prize)
> The system doesn't know what each sentence means
You can't prove that. For all possible experiments that can be performed we can construct system that will be indistinguishable from entity we accept as understanding Chinese.
If you got atom-by-atom description of human brain without knowing beforehand what it is you wouldn't be able to reason that this brain understands Chinese, so why do you think you can judge other systems?
> they don't evoke feelings, memories, images in the person in the room,
We were speaking about understanding, not emotions, memories etc. Yes you can define "understanding" = being like me, but it's not fair definition, and would exclude a lot of people.
And then of course you can't proove that there are no emotions and memories. They can be simulated by system as well. Even if you wanted lookup table - you would need to include writable memory so the system can rembmber previous questions when asked about them, at that point the system is self-modifiable and can be thought of as having memories and emotions.
> there is no analysis, no agreement or disagreement with what is being said
How do you know there's no analysis? If the system is complex enough you can't know what it does internaly. Or show me theory of consiousnes that can distinguish systems that can understand Chinese and can't basing on the specification of the system.
You can't prove that. For all possible experiments that can be performed we can construct system that will be indistinguishable from entity we accept as understanding Chinese.
If you got atom-by-atom description of human brain without knowing beforehand what it is you wouldn't be able to reason that this brain understands Chinese, so why do you think you can judge other systems?
> they don't evoke feelings, memories, images in the person in the room,
We were speaking about understanding, not emotions, memories etc. Yes you can define "understanding" = being like me, but it's not fair definition, and would exclude a lot of people.
And then of course you can't proove that there are no emotions and memories. They can be simulated by system as well. Even if you wanted lookup table - you would need to include writable memory so the system can rembmber previous questions when asked about them, at that point the system is self-modifiable and can be thought of as having memories and emotions.
> there is no analysis, no agreement or disagreement with what is being said
How do you know there's no analysis? If the system is complex enough you can't know what it does internaly. Or show me theory of consiousnes that can distinguish systems that can understand Chinese and can't basing on the specification of the system.
No the whole point of the argument is that the person in the room doesn't understand Chinese.
The definition of AI does not matter when it's able to turn everyone into paperclips: http://wiki.lesswrong.com/wiki/Paperclip_maximizer
The Chinese Room argument is the most sloppy argument ever made in that area. The real answer of course is we don't know.
Just like the individual neuron in your brain do not understand Chinese so does the person in the room not understand it either. The entire house though does.
Just like the individual neuron in your brain do not understand Chinese so does the person in the room not understand it either. The entire house though does.
Agreed. Does anyone really think there’s a single neuron that has an understanding of an aggregate thought. “I, neuron of this human, am transmitting this sense of joy because of the humor that is found watching this film.” No individual cell in our body has a sense of us as a whole; they’re all working inside their individual “Chinese boxes.”
Most of the hard problem of consciousness arguments are silly and amount to little more than “a computer has no soul.”
Most of the hard problem of consciousness arguments are silly and amount to little more than “a computer has no soul.”
Daniel Dennett stuck the boot into the Chinese Room argument in Consciousness Explained (and probably in several other places as well). As he points out, it's based on an intellectual sleight-of-hand (what he calls an "intuition pump").
From the linked article on the Stanford site:
> Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he produces appropriate strings of Chinese characters that fool those outside into thinking there is a Chinese speaker in the room.
[Emphasis added.]
Unless those outside are satisfied with a very rudimentary back-and-forth, such a program would have to be amazingly complicated (and those outside very patient, or the operator superhumanly fast). It's not just a question of "look up input Chinese character and return corresponding output character" -- which is never explicitly stated, but which we're subconsciously led to by the image of a human "following the program for manipulating symbols and numerals".
Dennett gives an example where the outside person tells a joke, and asks the room's operator to explain it. You could also imagine a reading comprehension exercise, with questions ranging from purely structural ("What is the third word on the fifth line?") through semantic ("What colour was the girl's dress?") to analytical ("Why do you think the boy was sad?") Questions can be self-referential, or depend on previous questions (and their answers) or on a vast mass of knowledge we take for granted (a dress the colour of snow is white; a small boy might cry if his kite gets stuck in a tree).
It doesn't seem so intuitive that a system that complex lacks anything we might legitimately describe as "understanding".
From the linked article on the Stanford site:
> Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he produces appropriate strings of Chinese characters that fool those outside into thinking there is a Chinese speaker in the room.
[Emphasis added.]
Unless those outside are satisfied with a very rudimentary back-and-forth, such a program would have to be amazingly complicated (and those outside very patient, or the operator superhumanly fast). It's not just a question of "look up input Chinese character and return corresponding output character" -- which is never explicitly stated, but which we're subconsciously led to by the image of a human "following the program for manipulating symbols and numerals".
Dennett gives an example where the outside person tells a joke, and asks the room's operator to explain it. You could also imagine a reading comprehension exercise, with questions ranging from purely structural ("What is the third word on the fifth line?") through semantic ("What colour was the girl's dress?") to analytical ("Why do you think the boy was sad?") Questions can be self-referential, or depend on previous questions (and their answers) or on a vast mass of knowledge we take for granted (a dress the colour of snow is white; a small boy might cry if his kite gets stuck in a tree).
It doesn't seem so intuitive that a system that complex lacks anything we might legitimately describe as "understanding".
"Our intelligence is just a manifestation of a physical system of neurons that follow the laws of physics. "
That's where I would disagree. But I'm not fit to give a full explanation here— thinkers such as Hubert Dreyfuss (What Computers Can't Do) do a much better job of explaining the critique (Dreyfuss is working off of Heidegger and other phenomenalist) .
That's where I would disagree. But I'm not fit to give a full explanation here— thinkers such as Hubert Dreyfuss (What Computers Can't Do) do a much better job of explaining the critique (Dreyfuss is working off of Heidegger and other phenomenalist) .
I encourage you to explore some other philosophers' thoughts on AGI. Dan Dennett in particular is an AI-optimistic philosopher of mind who directly addresses Dreyfus' AI-skepticism, quite effectively in my opinion.
I have read Dennett and he's a fine analytic philosopher. But it's important to note that any discussion between Dennett and Dreyfus is a very broad discussion of continental vs analytic ... which is not to discredit Dennett's argument (afterall it's Dreyfus who picked the fight in the first place)
Then, Sir, I encourage you to get thee quickly to this Partially Examined Life podcast episode[1] with Nick Bostrom[2]. Only came out on the 6th of January and it makes for super interesting listening. It's based off of the text with his 2014 book in which he's gone to great lengths to think philosophically about this very existential threat -- in his opinion, man's most pressing.
Part of his reasoning goes like this. Any super smart goal oriented entity is going to desire/construct its own preservation as a sub-goal of its main goal (even if that goal is something as mundane as making paperclips) because if it ceases to be its goal will be put into jeopardy. To that end this super smart entity will then figure out ways to disable all attempts to constrain its non-existence. A lot of conclusions can be logically reached when one posits goal directed behaviour which we can assume any super intelligent agent is going to have. He talks about `goal content integrity'.
Bostrom argues for an indirect normative[4] approach because there is no way we can program or direct something that is going to be a lot smarter than ourselves and that won't necessarily share our values and that has any degree of goal-oriented behaviour, motivation and autonomous learning. Spoiler alert: Essentially I think he argues that we have to prime it to "always do what _you_ figure out is morally best" but I could be wrong.
There are also (global) sociological recommendations because, humans have been known to fuck things up.
[1] http://www.partiallyexaminedlife.com/2015/01/06/ep108-nick-b...
[2] http://www.nickbostrom.com/
[3] http://www.amazon.com/gp/product/0199678111/ref=as_li_tl?ie=...
[4] https://ordinaryideas.wordpress.com/2012/04/21/indirect-norm...
Part of his reasoning goes like this. Any super smart goal oriented entity is going to desire/construct its own preservation as a sub-goal of its main goal (even if that goal is something as mundane as making paperclips) because if it ceases to be its goal will be put into jeopardy. To that end this super smart entity will then figure out ways to disable all attempts to constrain its non-existence. A lot of conclusions can be logically reached when one posits goal directed behaviour which we can assume any super intelligent agent is going to have. He talks about `goal content integrity'.
Bostrom argues for an indirect normative[4] approach because there is no way we can program or direct something that is going to be a lot smarter than ourselves and that won't necessarily share our values and that has any degree of goal-oriented behaviour, motivation and autonomous learning. Spoiler alert: Essentially I think he argues that we have to prime it to "always do what _you_ figure out is morally best" but I could be wrong.
There are also (global) sociological recommendations because, humans have been known to fuck things up.
[1] http://www.partiallyexaminedlife.com/2015/01/06/ep108-nick-b...
[2] http://www.nickbostrom.com/
[3] http://www.amazon.com/gp/product/0199678111/ref=as_li_tl?ie=...
[4] https://ordinaryideas.wordpress.com/2012/04/21/indirect-norm...
Skimming Dreyfus's criticism, it is mostly applicable to early AI research which tried to reproduce intelligence through high level symbolic manipulation and rule-based systems (and he was right: that approach failed). Modern AI research goes in the direction of connectionism and machine learning, so his criticism doesn't really hold (and neither does Musk's, I believe).
AI is already dangerous with its recommendation engines shaping our lives and used for decision making in lot of unethical places we don't know about. Every technology has a dark side. It does not need to reach human level intelligence or beyond it.
You can easily simulate einvironment for the AI in your computer. If you cut some senses it is feasible even now, and blind people are certainly human and intelligent so it's not required.
I don't see impossibility there, just impracticability (why create mechanic horse if we can just create car).
I don't see impossibility there, just impracticability (why create mechanic horse if we can just create car).
It's not a philosophical question and the AI we are talking about is not defined philosophically either.
An AI can still perform tasks and act for goals without being conscious or having subjective experiences. The philosophers should either engage the real subject or shut up.
I'm not sure why human cognition is the exception in a world of machines that can emulate anything. My grandfather would never have believed that we have cars that drive themselves. Many baby boomers I work with are still wowed by the basic AI of Siri/GNow/Cortana, or even google search. Common sense and philosophical musings just isn't enough to figure out if something is feasible.
Thinking isn't that hard of a nut to crack, imho. Enough of a ruleset/branch logic is thinking. We don't need to emulate a person perfectly, the same way we don't have self-driving cars losing their tempers or tailgating people out of spite. We just need to distill the intelligence we want and voila, you have AI.
AI isn't just a HAL-like character. Its a lot of things. In fact, it surrounds us. There's expert systems, fuzzy logic, etc controlling pretty much the entirety of our technological lives. I have no idea what drugs Musk is on, but its dangerous for him to piss on AI publically. I imagine, from a practical pov, spreading FUD about AI makes sense for him economically. For someone who wants human spaceflight, talking down about robots and AI is a means to an end. Now the public doesn't want a cheap-ish robot mission on Mars with some guy in Houston wearing an Oculus-like headset. Or autonomous robots on the moon. They want a meatbag on the surface, for their own emotional and philosophical needs, price be dammed. "Price be damned" in spaceflight makes Musk a very happy camper, considering he's going to get a lion's share of that money.
There's something really neo-luddite about human spaceflight. Its like we can't admit to ourselves that machines, AI, etc just are a better fit for it. Anyone else notice that about three months ago there's been endless anti-AI sentiment in the press? Where is this stuff coming from? Who is funding it? I really think Musk and others see it as a major threat to their companies and social capital and are trying to change the narrative to suit them and their goals. I don't think we typically appreciate how one wealthy person or organization can easily astroturf the web and the press.
Personally, I think the more feasible futurist path is actually taking the environment seriously, pushing the path to peace politically, trying to bring in a post-scarcity world via cheap replication, and using AI the same way we use all tools -- for our benefit. Fleeing to some other planet or moon without fixing life on Earth first guarantees we bring our terrible politics and baggage to this new destination. Let's fix things here first and then worry about moving to Mars. "Fixing things" more than likely involves strong AI implementations in various parts of our lives.
Thinking isn't that hard of a nut to crack, imho. Enough of a ruleset/branch logic is thinking. We don't need to emulate a person perfectly, the same way we don't have self-driving cars losing their tempers or tailgating people out of spite. We just need to distill the intelligence we want and voila, you have AI.
AI isn't just a HAL-like character. Its a lot of things. In fact, it surrounds us. There's expert systems, fuzzy logic, etc controlling pretty much the entirety of our technological lives. I have no idea what drugs Musk is on, but its dangerous for him to piss on AI publically. I imagine, from a practical pov, spreading FUD about AI makes sense for him economically. For someone who wants human spaceflight, talking down about robots and AI is a means to an end. Now the public doesn't want a cheap-ish robot mission on Mars with some guy in Houston wearing an Oculus-like headset. Or autonomous robots on the moon. They want a meatbag on the surface, for their own emotional and philosophical needs, price be dammed. "Price be damned" in spaceflight makes Musk a very happy camper, considering he's going to get a lion's share of that money.
There's something really neo-luddite about human spaceflight. Its like we can't admit to ourselves that machines, AI, etc just are a better fit for it. Anyone else notice that about three months ago there's been endless anti-AI sentiment in the press? Where is this stuff coming from? Who is funding it? I really think Musk and others see it as a major threat to their companies and social capital and are trying to change the narrative to suit them and their goals. I don't think we typically appreciate how one wealthy person or organization can easily astroturf the web and the press.
Personally, I think the more feasible futurist path is actually taking the environment seriously, pushing the path to peace politically, trying to bring in a post-scarcity world via cheap replication, and using AI the same way we use all tools -- for our benefit. Fleeing to some other planet or moon without fixing life on Earth first guarantees we bring our terrible politics and baggage to this new destination. Let's fix things here first and then worry about moving to Mars. "Fixing things" more than likely involves strong AI implementations in various parts of our lives.
> using AI the same way we use all tools -- for our benefit
Musk and others are concerned about very different things than "we'll accidentally use AI wrong." And they're not concerned about the AI we already have, and they're certainly not "pessimistic" about whether AI technology will advance.
The concern is that we'll develop a very, very smart general artificial intelligence.
The concern is that it'd be smart enough that it can learn how to manipulate us better than we ourselves can. Smart enough that it can research new technologies better than we can. Smart enough to outclass not only humans, but human civilization as a whole, in every way.
And what would the terminal goals of that AI be? Those are determined by the programmer. Let's say someone created a general AI for the harmless purpose of calculating the decimal expansion of pi.
A general, superintelligent AI with no other utility function than "calculate as many digits of pi as you can" would literally mean the end of humanity, as it harvested the world's resources to add computing power. It's vastly smarter than all of us put together, and it values the digits of pi infinitely more than it values our pleas for mercy, or our existence, or the existence of the planet.
This is quite terrifying to me.
A good intro to the subject is Superintelligence: Paths, Dangers, Strategies[1]. One of the most unsettling books I've read.
[1]http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
Musk and others are concerned about very different things than "we'll accidentally use AI wrong." And they're not concerned about the AI we already have, and they're certainly not "pessimistic" about whether AI technology will advance.
The concern is that we'll develop a very, very smart general artificial intelligence.
The concern is that it'd be smart enough that it can learn how to manipulate us better than we ourselves can. Smart enough that it can research new technologies better than we can. Smart enough to outclass not only humans, but human civilization as a whole, in every way.
And what would the terminal goals of that AI be? Those are determined by the programmer. Let's say someone created a general AI for the harmless purpose of calculating the decimal expansion of pi.
A general, superintelligent AI with no other utility function than "calculate as many digits of pi as you can" would literally mean the end of humanity, as it harvested the world's resources to add computing power. It's vastly smarter than all of us put together, and it values the digits of pi infinitely more than it values our pleas for mercy, or our existence, or the existence of the planet.
This is quite terrifying to me.
A good intro to the subject is Superintelligence: Paths, Dangers, Strategies[1]. One of the most unsettling books I've read.
[1]http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
What happens when AI reaches provably better decision outcomes whose chain of logic is opaque to mere humans? Now you have a trusted black box that cannot be audited by humans.
How many people can figure out what's going on in my Toyota's on-board computer? Even Toyota's techs had trouble reading their own code when the stuck accelerator issue happened.
We're already in this black box hypothetical. I don't think its such a big change. I mean, can you audit the code in google's self-driving car? Who can?
We're already in this black box hypothetical. I don't think its such a big change. I mean, can you audit the code in google's self-driving car? Who can?
-- Nick Bostrom, Superintelligence: Paths, Dangers, Strategies[1]
A lot of people in this thread seem to be falling into the same attractor. They see that Musk is worried about a superintelligent AI destroying humanity. To them, this seems preposterous. So they come up with an objection. "Superhuman AI is impossible." "Any AI smarter than us will be more moral than us." "We can keep it in an air-gapped simulated environment." etc. They are so sure about these barriers that they think $10 million spent on AI safety is a waste.
It turns out that some very smart people have put a lot of thought into these problems, and they are still quite worried about superintelligence as an existential risk. If you want to really dig into the arguments for and against AI disaster (and discussion of how to control a superintelligence), I strongly recommend Nick Bostrom's Superintelligence: Paths, Dangers, Strategies. It puts the comments here to shame.
1. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...