Pentagon: A Human Will Always Decide When a Robot Kills You(wired.com)
wired.com
Pentagon: A Human Will Always Decide When a Robot Kills You
http://www.wired.com/dangerroom/2012/11/human-robot-kill/?utm_source=twitter&utm_medium=socialmedia&utm_campaign=twitterclickthru
20 comments
Many are now calling for this level of international law:
http://theglobeandmail.com/news/world/ban-urged-on-killer-ro...
http://theglobeandmail.com/news/world/ban-urged-on-killer-ro...
One possibility that the article seems to ignore is that in the future, robots may be explicitly programmed to follow laws of war and rules of engagement. There's already been some research into this [1], and some people argue that autonomous robots could eventually be better than humans at following international law in combat. It does seem like a stretch given the present state of AI, but in the future there might be treaties that specify the algorithms that autonomous systems have to implement if they're going to be deployed to combat situations.
[1] http://www.cc.gatech.edu/ai/robot-lab/online-publications/fo...
[1] http://www.cc.gatech.edu/ai/robot-lab/online-publications/fo...
Where does the battlefield stop when drones allow soldiers to be remote?
If a drone pilot on active duty works and lives in a civilian population, are those civilians being used as human shields? How does that sit with the Geneva Convention? Is a US based pilot, who goes home of a night, counted to be on the front-line and unwittingly using their family as human shields?
I'd think the use of drones extends the bounds of the battlefield, making drone pilots fair game. Probably a moot point when fighting a adversary with limited range, but it might be quite important in a more equal contest.
If a drone pilot on active duty works and lives in a civilian population, are those civilians being used as human shields? How does that sit with the Geneva Convention? Is a US based pilot, who goes home of a night, counted to be on the front-line and unwittingly using their family as human shields?
I'd think the use of drones extends the bounds of the battlefield, making drone pilots fair game. Probably a moot point when fighting a adversary with limited range, but it might be quite important in a more equal contest.
A human may call the kill order, but that order is based upon information the robots/computer/machine gives them.
How many people have died from bad code, thats more worrying.
How many people have died from bad code, thats more worrying.
I think if we had actual people on the ground engaging the enemy using traditional warfare, it's pretty much a given there would be more casualties on both sides. From that standpoint drones are preferable --they offer an asymmetrical aspect to the war, so it might seen it as unfair, but that's beside the point as IEDs are seen as unfair as well. With UAVs Fewer friendlies die, and fewer civilians on the enemy side die due to the more precise aspect of the engagement. Even with the misses, drone engagement results in fewer casualties for both sides.
Look at it this way, if the US had had drones during the VN conflict, the US would not have incurred 50,000 casualties and the VC would not have suffered ~2,000,000 casualties. We'd have gone after the Võ Nguyên Giáps of the conflict.
Look at it this way, if the US had had drones during the VN conflict, the US would not have incurred 50,000 casualties and the VC would not have suffered ~2,000,000 casualties. We'd have gone after the Võ Nguyên Giáps of the conflict.
Is there any evidence that the information provided by any technology for a drone attack has been inaccurate because of a bug or bad code?
It's not like the government is using machine learning to categorize likely terrorists and then blowing them up in their homes.
It's not like the government is using machine learning to categorize likely terrorists and then blowing them up in their homes.
1. is there any evidence supporting the existence of a bug free system?
2. you're probably over-simplifying the actual process in which we algorithmically identify and eliminate terror threats. which i'd bet for all intents and purposes is just as scary.
my point here is that you, like you identified of your parent's post, are operating under quite a few assumptions.
2. you're probably over-simplifying the actual process in which we algorithmically identify and eliminate terror threats. which i'd bet for all intents and purposes is just as scary.
my point here is that you, like you identified of your parent's post, are operating under quite a few assumptions.
Isn't that nice. This isn't the most concerning aspect about the use of drones though. The concerning part is the extension of the definition of "battlefield".
Currently the definition of what is a battlefield is little different than "wherever we decide to put a drone". However, this leaves a lot to be desired from a rules-of-war and civil liberties perspective. The thought that merely visiting some city in the developing world and hanging around with the wrong folks could result in you dying in a battlefield via remote drone attack is more than a little disturbing, especially when compared against all of the normal heuristics a sensible human with good-intentions would typically use to avoid being involved in a war and especially being killed in battle.
Certainly there are risks and downsides to operating under rules-of-war which are too limiting or too strict but the solution is not to throw everything out and create effectively unlimited power with almost no safeguards, limitations, or accountability. And that is the road we seem to be heading down. As history has shown, power without accountability and limitations is never a good idea and will inevitably lead to intentional abuse.
Currently the definition of what is a battlefield is little different than "wherever we decide to put a drone". However, this leaves a lot to be desired from a rules-of-war and civil liberties perspective. The thought that merely visiting some city in the developing world and hanging around with the wrong folks could result in you dying in a battlefield via remote drone attack is more than a little disturbing, especially when compared against all of the normal heuristics a sensible human with good-intentions would typically use to avoid being involved in a war and especially being killed in battle.
Certainly there are risks and downsides to operating under rules-of-war which are too limiting or too strict but the solution is not to throw everything out and create effectively unlimited power with almost no safeguards, limitations, or accountability. And that is the road we seem to be heading down. As history has shown, power without accountability and limitations is never a good idea and will inevitably lead to intentional abuse.
> hanging around with the wrong folks could result in you dying in a battlefield via remote drone attack is more than a little disturbing
Being a minor (16 years old) and a US citizen (born in Denver) won't protect you from "having the wrong father", as Abdulrahman_al-Aulaqi can tell you[1].
It's not reassuring at all to me to know that a human, rather than a robot, is making the decision, as long as I don't know (or trust) the criteria that that human is using.
[1] (Except he can't, because he was assassinated last year, far from any "battlefield" - or even any country that the US is at war with, for that matter.)
Being a minor (16 years old) and a US citizen (born in Denver) won't protect you from "having the wrong father", as Abdulrahman_al-Aulaqi can tell you[1].
It's not reassuring at all to me to know that a human, rather than a robot, is making the decision, as long as I don't know (or trust) the criteria that that human is using.
[1] (Except he can't, because he was assassinated last year, far from any "battlefield" - or even any country that the US is at war with, for that matter.)
"and a US citizen (born in Denver) "
I've never understood the emphasis on this point.
You're either for or against drone attacks on presumed terrorists, I can't understand how US citizenship plays any role in it. If his father was involved in terrorism and you think drone strikes are OK, why would you have extra sympathy for him because he's a US citizen?
I don't think the US citizen aspect is worth mentioning, in fact I think it just makes people who are against drones seem a little unhinged. By the way I am against the use of drones (on Americans or otherwise).
I've never understood the emphasis on this point.
You're either for or against drone attacks on presumed terrorists, I can't understand how US citizenship plays any role in it. If his father was involved in terrorism and you think drone strikes are OK, why would you have extra sympathy for him because he's a US citizen?
I don't think the US citizen aspect is worth mentioning, in fact I think it just makes people who are against drones seem a little unhinged. By the way I am against the use of drones (on Americans or otherwise).
> I've never understood the emphasis on this point.
All human lives are equally important from a moral perspective [1].
But from the perspective of a country's government -- any country, not just the US -- their own citizens' lives are the most important.
Imagine a man kills a woman -- a random stranger he met on the street. It's a terrible crime.
Now imagine a man kills a woman -- his daughter. Most people's gut reaction is that this is a worse crime than the previous example, because a parent he has a moral duty to protect his child.
Likewise, a country is supposed to protect its citizens -- so killing its own requires a stronger justification than killing non-citizens.
I'm not going to take the discussion off-topic by taking a position on the circumstances under which it's justifiable for a country to kill its own citizens or non-citizens. I'm just saying that a country's justification for killing its own citizens needs to be at least as strong -- and usually stronger -- than its justification for killing non-citizens.
[1] This paragraph is completely off-topic. "All human lives are equally important" sounds like a good and simple axiom for a value system. But it's interesting to try to think of situations where any of several people could be sacrificed to prevent some great evil, and the group figures out reasons to sacrifice some person or people with certain characteristics in preference to others. For example, "save the women and children first" is often said in disaster movies -- which implies their lives are somehow more valuable. Based on this anecdotal evidence, it appears that real human value systems often don't follow this axiom.
All human lives are equally important from a moral perspective [1].
But from the perspective of a country's government -- any country, not just the US -- their own citizens' lives are the most important.
Imagine a man kills a woman -- a random stranger he met on the street. It's a terrible crime.
Now imagine a man kills a woman -- his daughter. Most people's gut reaction is that this is a worse crime than the previous example, because a parent he has a moral duty to protect his child.
Likewise, a country is supposed to protect its citizens -- so killing its own requires a stronger justification than killing non-citizens.
I'm not going to take the discussion off-topic by taking a position on the circumstances under which it's justifiable for a country to kill its own citizens or non-citizens. I'm just saying that a country's justification for killing its own citizens needs to be at least as strong -- and usually stronger -- than its justification for killing non-citizens.
[1] This paragraph is completely off-topic. "All human lives are equally important" sounds like a good and simple axiom for a value system. But it's interesting to try to think of situations where any of several people could be sacrificed to prevent some great evil, and the group figures out reasons to sacrifice some person or people with certain characteristics in preference to others. For example, "save the women and children first" is often said in disaster movies -- which implies their lives are somehow more valuable. Based on this anecdotal evidence, it appears that real human value systems often don't follow this axiom.
> "All human lives are equally important" sounds like a good and simple axiom for a value system.
Sounds like a useful simplification at most.
Sounds like a useful simplification at most.
> a useful simplification at most.
As I footnoted, that's really what it is. But I used it to start my comment, because the parent's viewpoint may implicitly rely on it.
As I footnoted, that's really what it is. But I used it to start my comment, because the parent's viewpoint may implicitly rely on it.
Citizens are entitled to due process. There are probably a bunch of other reasons i'm overlooking, but I get kind of uptight about the government ignoring the bill of rights.
The reason it's important is that regardless of whatever else the purpose of the laws of the US are to protect US citizens and maintain their rights. That's a fundamental, founding aspect of our country and heavily embedded within the constitution.
Here we have a situation where, to put it in the least glamorous light, a US citizen may be summarily executed for little more than being outside of US soil. That's a bit harsh but it's compatible with the track record so far. And if a US citizen has no protections outside of US territory then that implies non-US citizens have even less protections. To say that such a scenario is distressing is to put it mildly.
What happens when, not if, other countries build up their drone fleets and engage in similar low-level warfare throughout the world? Will the developing world become a de facto 24/7 battlefield contested over by the developed countries? Will the lives of most of the world's citizens be subject to nothing more than the whims of some bureaucrat or soldier in a richer, more powerful country?
Here we have a situation where, to put it in the least glamorous light, a US citizen may be summarily executed for little more than being outside of US soil. That's a bit harsh but it's compatible with the track record so far. And if a US citizen has no protections outside of US territory then that implies non-US citizens have even less protections. To say that such a scenario is distressing is to put it mildly.
What happens when, not if, other countries build up their drone fleets and engage in similar low-level warfare throughout the world? Will the developing world become a de facto 24/7 battlefield contested over by the developed countries? Will the lives of most of the world's citizens be subject to nothing more than the whims of some bureaucrat or soldier in a richer, more powerful country?
> If his father was involved in terrorism and you think drone strikes are OK, why would you have extra sympathy for him because he's a US citizen?
As of April 2010, the open, stated objective of the CIA was to kill Aulaqi's father (and others on President Obama's "hit list") and, if and only if he could not be killed, to arrest him, charge him formally, and bring him to trial.
In other words, "if we can kill him, we don't have to prove he was actually guilty."
I don't support the drone strikes against non-US citizens, but the idea that a single executive can issue capital punishment on a US citizen[1] without a trial is downright unnerving. Imagine if police officers wielded that same level of power - it's no different.
[1] The US Constitution does not provide the same levels of protection to non-Citizens, which is why drone-strikes on foreigners is still disgusting but not as terrifying (from the perspective of a US citizen).
As of April 2010, the open, stated objective of the CIA was to kill Aulaqi's father (and others on President Obama's "hit list") and, if and only if he could not be killed, to arrest him, charge him formally, and bring him to trial.
In other words, "if we can kill him, we don't have to prove he was actually guilty."
I don't support the drone strikes against non-US citizens, but the idea that a single executive can issue capital punishment on a US citizen[1] without a trial is downright unnerving. Imagine if police officers wielded that same level of power - it's no different.
[1] The US Constitution does not provide the same levels of protection to non-Citizens, which is why drone-strikes on foreigners is still disgusting but not as terrifying (from the perspective of a US citizen).
>[1] The US Constitution does not provide the same levels of
> protection to non-Citizens, which is why drone-strikes on
> foreigners is still disgusting but not as terrifying (from
> the perspective of a US citizen).
For a scholarly article on this topic, see: http://scholarship.law.georgetown.edu/cgi/viewcontent.cgi?ar...
> Imagine if police officers wielded that same level of power - it's no different.
Seems to me that there are plenty of police officers who try through the claim of self defense.
Seems to me that there are plenty of police officers who try through the claim of self defense.
That is a good ethical point but it's normally brought up because it's a legal issue. It's (even more) illegal for the US government to be assassinating their own citizens. Only international law would have anything to say about assassinating non-US citizen civilians.
Because US citizens have rights under the US Constitution?
When you go to another country and openly wage war upon the US, you make yourself a military target. You don't get to wrap yourself in the constitution and enjoy all the procedural protections of the criminal justice system under such circumstances. If Anwar al -Awlaki had felt his life was in danger, he could have just turned himself in to a US consulate and asked for his legal rights.
As for his son, I blame the father for taking him with him to live in an Al-Qaeda stronghold, and effectively using him as a human shield.
As for his son, I blame the father for taking him with him to live in an Al-Qaeda stronghold, and effectively using him as a human shield.
> When you go to another country and openly wage war upon the US, you make yourself a military target.
His main activity was recording videos in which he expressed his ideas. If he was thought to have committed other crimes, he had the right to be tried for them.
> As for his son, I blame the father for taking him with him to live in an Al-Qaeda stronghold
That didn't happen. IIRC, he was killed at a dinner party. We're not even certain who it was the US was trying to kill during that strike.
His main activity was recording videos in which he expressed his ideas. If he was thought to have committed other crimes, he had the right to be tried for them.
> As for his son, I blame the father for taking him with him to live in an Al-Qaeda stronghold
That didn't happen. IIRC, he was killed at a dinner party. We're not even certain who it was the US was trying to kill during that strike.
[deleted]
The drone strike that killed the kid was targeting Ibrahim al-Banna (and it succeeded in killing him). They almost certainly did not expect that the kid would be there, considering that he lived over 100 miles away far from where that drone struck.
But you don't have to openly do anything. Military aged males in specific areas are presumed militants. Where those areas are is classified.
> If Anwar al -Awlaki had felt his life was in danger, he could have just turned himself in to a US consulate and asked for his legal rights
Hahahahaha, that's a good one.
> If Anwar al -Awlaki had felt his life was in danger, he could have just turned himself in to a US consulate and asked for his legal rights
Hahahahaha, that's a good one.
The guy made extensive use of the internet to broadcast his views. Not once did he say 'I reject these accusations and seek a trial/political asylum in some other country.' If you spend all your time glorifying violence and martyrdom, then you can have no complaints when people take you at your word.
There are also legal arguments in favor of this position, but past experiences suggests I'll do a lot of typing and linking to no particular end, so I'll confine myself to quoting Justice Holmes: the constitution is not a suicide pact.
There are also legal arguments in favor of this position, but past experiences suggests I'll do a lot of typing and linking to no particular end, so I'll confine myself to quoting Justice Holmes: the constitution is not a suicide pact.
>When you go to another country and openly wage war upon the US, you make yourself a military target.
This is understandable - if Americans are being attacked, their attackers should be killed. Figure out who they are later. But in this case there was no immediate danger. And to steal from Stalin, "how many divisions does al-Awlaki have?" I would love to hear the evidence against al-Awlaki, but sadly the trial was dispensed with, not to mention the subpoena.
This is understandable - if Americans are being attacked, their attackers should be killed. Figure out who they are later. But in this case there was no immediate danger. And to steal from Stalin, "how many divisions does al-Awlaki have?" I would love to hear the evidence against al-Awlaki, but sadly the trial was dispensed with, not to mention the subpoena.
What is the definition of a "human shield" there, being in the vicinity of people who shouldn't be killed?
Humans make mistakes. We should really put that kind of decision making in the hands of an artificial intelligence.
it's gonna happen anyway whether the pentagon says something or not. not that i understand why anyone cares what the pentagon says.
I was saying that tongue in cheek, but yeah, it will happen.
At some point the brainiacs in the war department will realize that you can never find enough soldiers to police everyone -- if they haven't already realized this. The only way to solve that is make a machine more efficient than people.
Our ally on the inside, potentially, are soldiers like the general in Wargames played by Barry Corbin, who don't trust the machine over a human soldier.
At some point the brainiacs in the war department will realize that you can never find enough soldiers to police everyone -- if they haven't already realized this. The only way to solve that is make a machine more efficient than people.
Our ally on the inside, potentially, are soldiers like the general in Wargames played by Barry Corbin, who don't trust the machine over a human soldier.
I'm not entirely sure I believe this. Yes, for predators and such drones it makes sense. But the military already has weapons like the sea whiz that seek, aim, and fire all on their own[1]. No human pulling the trigger or picking out targets. Granted, the Sea Whiz is anti-missile. But it's only a matter of time until it becomes self aware and targets whatever it wants....
[1] http://en.wikipedia.org/wiki/Sea_whiz
edit: that gun looks fantastic: http://www.youtube.com/watch?v=iAw82h-IhdQ&feature=fvwre...
[1] http://en.wikipedia.org/wiki/Sea_whiz
edit: that gun looks fantastic: http://www.youtube.com/watch?v=iAw82h-IhdQ&feature=fvwre...
What happens if a foreign power decides to unleash 1000 armed mini-drones in NYC as revenge?
How exactly will the US stop them?
All that TSA money could have been used to research that problem.
Are we going to have drone dogfights? Because the next step is to remove the need for human operators for most automation like evasion, so frequency jamming is out. I guess EMI pulse is the only way but they can be shielded too.
How exactly will the US stop them?
All that TSA money could have been used to research that problem.
Are we going to have drone dogfights? Because the next step is to remove the need for human operators for most automation like evasion, so frequency jamming is out. I guess EMI pulse is the only way but they can be shielded too.
> What happens if a foreign power decides to unleash 1000 armed mini-drones in NYC as revenge?
at that point, it'd be all out open war. The only reason the US can do these operations in foreign lands is that no other country could really retaliate (and survive).
at that point, it'd be all out open war. The only reason the US can do these operations in foreign lands is that no other country could really retaliate (and survive).
>>so frequency jamming is out
I don't know about that, they would still have to communicate with eachother/use radar to be aware of their surroundings/use gps to know where they are. Plenty of avenues for jamming/spoofing.
I don't know about that, they would still have to communicate with eachother/use radar to be aware of their surroundings/use gps to know where they are. Plenty of avenues for jamming/spoofing.
Unless you happen to be standing too close to someone a human decides to kill with a drone, of course.
Wow, that will make it feel much better if they off you, how warm and fuzzy. They should make a hallmark card "Seasons Greetings, if you get taken out, you can count on us to ensure that a human being will decide when a robot kills you. Sincerely, The Pentagon."
Coincidental that on the same day there is an HN frontpage story about Google Spanner database.
TL;DR humans are too slow
...think this ends well for slow humans?
TL;DR humans are too slow
...think this ends well for slow humans?
well, this has been not true for a long time now.
Mines.
Anti-personnel, anti-tank, whatever. Those are killing automata, some very simple, some quite smart - not that easy to kill a tank with high probability.
so, yeah, there will be next gen mines, combine a rifle with some sensors and a field of fire. voila, a stand off weapon covering your flank, nothing comes through this alley for a while.
Mines.
Anti-personnel, anti-tank, whatever. Those are killing automata, some very simple, some quite smart - not that easy to kill a tank with high probability.
so, yeah, there will be next gen mines, combine a rifle with some sensors and a field of fire. voila, a stand off weapon covering your flank, nothing comes through this alley for a while.
And Iraq has weapons of mass destruction.
Until a robot decides not to listen...
You don't need AI to make the assertion literally false. the pentagon isn't giving up on the concept of "collateral damage".
Impose a law and someone will break it.
"And that's our promise to you, here at the Pentagon."
Shouldn't something this important be implemented at a higher level?
I'd think there ought to be a new Geneva convention about the rules of robot warfare. Get all the countries capable of making deadly robots to agree on what kinds of tactics are okay and the chain of command/responsibility issue addressed by this memo.
Of course there's no guarantee that all countries will sign it, or that those that do will keep to their promises, or that it will regulate terrorists.
I do approve of the move; it's definitely a step in the right direction. It's better than having no rules on the issue, or rules saying that software should have the power to make those decisions.
But the government should be taking a stronger stance on the issue.