Palantir demos AI to fight wars but says it will be ethical(vice.com)
vice.com
Palantir demos AI to fight wars but says it will be ethical
https://www.vice.com/en/article/qjvb4x/palantir-demos-ai-to-fight-wars-but-says-it-will-be-totally-ethical-dont-worry-about-it
9 comments
This headline (and Palantir and our country) ignores the large swath of the population who doesn't find war ethical in the first place.
Oh I'd say Palantir and the U.S. military understand quite well how many people, especially our own soldiers, find what we're doing unethical, which is why they're so keen to replace human soldiers with amoral robots.
What happens when: You are walking a beautiful forest with a beautiful Ukranian girl, and somebody drops a war on you, nobody so much asked for it...That until the last minute they did not believe was going to happen?
Ignoring the obvious ethical questions, there's also no real assurance that this won't be vulnerable to prompt injection. Is it going to do image analysis? Is it going to work with 3rd-party text? Can it do web searches?
And they want this crap to be able to automatically execute workflows without human intervention.
> What AIP does not do is walk through how it plans to deal with the various pernicious problems of LLMs and what the consequences might be in a military context. AIP does not appear to offer solutions to those problems beyond “frameworks” and “guardrails” it promises will make the use of military AI “ethical” and “legal.”
Utterly, utterly ridiculous. But it's OK because they said it's "ethical."
The computing industry has gone completely off the rails with LLMs; we spent a month or two debating hypothetical sci-fi questions until everyone got bored of them, and now nobody is thinking about ethics or security at all for any of their products. It would be embarrassing if it wasn't so horrifying.
What we've learned from this is that "how does the AI escape the box?" is a fun thought experiment, but "is this technology inherently vulnerable to extremely basic malicious input attacks?" is a boring question that's not fun to debate and might have implications for the stuff we're actually building, so we're just not going to have that conversation. Or if we do have that conversation, we're definitely not going to delay anything we're building until we have an answer to that question. We're just going to say it's an ongoing issue that will be fixed sometime in the future, probably.
It's not reasonable to ask us to solve that problem before we let an LLM control a military drone. /s
But every single journalist or writer who covers this stuff should be pointing out that so far, the industry has basically zero idea of how to actually secure an LLM in production. And that's not even getting into "what happens when your Chat AI hallucinates an answer to a question that's wrong and then orders a drone strike off of that hallucination?" Figuring out how to build an LLM agent that can't be hacked is the baseline, and the industry hasn't even passed that point yet.
And they want this crap to be able to automatically execute workflows without human intervention.
> What AIP does not do is walk through how it plans to deal with the various pernicious problems of LLMs and what the consequences might be in a military context. AIP does not appear to offer solutions to those problems beyond “frameworks” and “guardrails” it promises will make the use of military AI “ethical” and “legal.”
Utterly, utterly ridiculous. But it's OK because they said it's "ethical."
The computing industry has gone completely off the rails with LLMs; we spent a month or two debating hypothetical sci-fi questions until everyone got bored of them, and now nobody is thinking about ethics or security at all for any of their products. It would be embarrassing if it wasn't so horrifying.
What we've learned from this is that "how does the AI escape the box?" is a fun thought experiment, but "is this technology inherently vulnerable to extremely basic malicious input attacks?" is a boring question that's not fun to debate and might have implications for the stuff we're actually building, so we're just not going to have that conversation. Or if we do have that conversation, we're definitely not going to delay anything we're building until we have an answer to that question. We're just going to say it's an ongoing issue that will be fixed sometime in the future, probably.
It's not reasonable to ask us to solve that problem before we let an LLM control a military drone. /s
But every single journalist or writer who covers this stuff should be pointing out that so far, the industry has basically zero idea of how to actually secure an LLM in production. And that's not even getting into "what happens when your Chat AI hallucinates an answer to a question that's wrong and then orders a drone strike off of that hallucination?" Figuring out how to build an LLM agent that can't be hacked is the baseline, and the industry hasn't even passed that point yet.
Agreed. This is just ludicrous. People just keep saying "but think of the benefits!" and then you repeatedly start seeing things like this and you have to ask yourself... when it's all said and done are you going to like the world you wind up with?
> And they want this crap to be able to automatically execute workflows without human intervention.
I don't think that's what I saw in the tech demo (sales video?). At least there, a human made the actual decisions, and the LLM did things like:
- notify of enemy units
- show the user what reconnaissance units are available in the area
- come up with possible courses of action
- generate routes (probably by calling some other model that does this)
When the user tells the LLM to send the recon drone, send the course of action up the chain of command, submits operational plans, etc., it shows what was the specific action performed. And for the example operation they had, sending a strike team, that, at least for now, still has to be done by real human beings.
Not to say that this isn't anything to be worried about even if the LLM is sandboxed and well-guardrailed from making real-world decisions on its own; you should definitely be just as worried as humans augmented with the ability to more efficiently kill others, even if it's only with what amounts to an AI secretary.
I don't think that's what I saw in the tech demo (sales video?). At least there, a human made the actual decisions, and the LLM did things like:
- notify of enemy units
- show the user what reconnaissance units are available in the area
- come up with possible courses of action
- generate routes (probably by calling some other model that does this)
When the user tells the LLM to send the recon drone, send the course of action up the chain of command, submits operational plans, etc., it shows what was the specific action performed. And for the example operation they had, sending a strike team, that, at least for now, still has to be done by real human beings.
Not to say that this isn't anything to be worried about even if the LLM is sandboxed and well-guardrailed from making real-world decisions on its own; you should definitely be just as worried as humans augmented with the ability to more efficiently kill others, even if it's only with what amounts to an AI secretary.
It's somewhat subtle and easy to miss, but one of the feature points raised in the video about the AIP action graph is that users can "govern LLMs, defining when they can act alone and where they need human input."
So while the demo does mostly show back-and-forth between the LLM and a user, I'm still somewhat worried about what the limits of that "action graph" are going to be and how much of that back-and-forth in the demo is actually mandatory, because I do get the impression that part of the feature set is going to be automated responses. Maybe (hopefully) I'm wrong.
But even if that action graph is taken away and the chat back-and-forth is the only interface, then consider one of your examples:
> When the user tells the LLM to send the recon drone
This is still a giant problem. Can the LLM send the recon drone and it's just instructed to wait for the user to tell it to, or (preferably) does the LLM have zero control over the recon drone and is the user sending it through a separate interface that is not connected to the LLM at all? To me from the demo, it looks to me like a user can just ask an LLM to perform that action. That's a big problem, the product shouldn't launch with that feature.
Because if the LLM has access to that API and if there's any opportunity for malicious input (image analysis/OCR, web searches, etc...), then that input can cause the LLM to skip authorization and send the recon drone without human approval.
> you should definitely be just as worried as humans augmented with the ability to more efficiently kill others, even if it's only with what amounts to an AI secretary.
You're right, and it feels a little weird to have the primary criticism be "they're not securely killing people with AI." But for every action you've described above I have to ask, "can a 3rd-party reprogram the AI and get it to lie about the result of that action or get it to take that action without human intervention?"
I'm worried about military drone strikes from an ethical perspective, I'm worried about AI security within the military from a "holy crap, who authorized this and why haven't they been fired?" perspective. I'm not sure I've seen a single large commercial deployment of an LLM wired to real-world systems where I felt like the company building that product had rock-solid security; so I'm doubtful that Plantir is going to be the first one.
So while the demo does mostly show back-and-forth between the LLM and a user, I'm still somewhat worried about what the limits of that "action graph" are going to be and how much of that back-and-forth in the demo is actually mandatory, because I do get the impression that part of the feature set is going to be automated responses. Maybe (hopefully) I'm wrong.
But even if that action graph is taken away and the chat back-and-forth is the only interface, then consider one of your examples:
> When the user tells the LLM to send the recon drone
This is still a giant problem. Can the LLM send the recon drone and it's just instructed to wait for the user to tell it to, or (preferably) does the LLM have zero control over the recon drone and is the user sending it through a separate interface that is not connected to the LLM at all? To me from the demo, it looks to me like a user can just ask an LLM to perform that action. That's a big problem, the product shouldn't launch with that feature.
Because if the LLM has access to that API and if there's any opportunity for malicious input (image analysis/OCR, web searches, etc...), then that input can cause the LLM to skip authorization and send the recon drone without human approval.
> you should definitely be just as worried as humans augmented with the ability to more efficiently kill others, even if it's only with what amounts to an AI secretary.
You're right, and it feels a little weird to have the primary criticism be "they're not securely killing people with AI." But for every action you've described above I have to ask, "can a 3rd-party reprogram the AI and get it to lie about the result of that action or get it to take that action without human intervention?"
I'm worried about military drone strikes from an ethical perspective, I'm worried about AI security within the military from a "holy crap, who authorized this and why haven't they been fired?" perspective. I'm not sure I've seen a single large commercial deployment of an LLM wired to real-world systems where I felt like the company building that product had rock-solid security; so I'm doubtful that Plantir is going to be the first one.
Yes, I think dispatching the drone could be done in a safer way. E.g., by sending the user some button to click on to send the command the LLM suggests to the drone. They have the confirmation messages saying what action was performed, but that's still problematic for the reasons you mentioned.
“hypothetical sci-fi”
I don’t think it’s fair to dismiss the concerns entirely (well the tone sounds a bit dismissive even if you didn’t explicitly say it’s not a problem) based on the fact that it hasn’t happened yet (https://xkcd.com/2278/) and that the topic has been covered by science fiction before (though it isn’t normally shown in a very realistic way)
I don’t think it’s fair to dismiss the concerns entirely (well the tone sounds a bit dismissive even if you didn’t explicitly say it’s not a problem) based on the fact that it hasn’t happened yet (https://xkcd.com/2278/) and that the topic has been covered by science fiction before (though it isn’t normally shown in a very realistic way)
It is a little bit dismissive, which I apologize for, but my intent is not to mock the people worried about AGI as much as to point out that the conversations around AGI in mainstream contexts tend to be reductive, inaccurate, or oversimplified, and I'm frustrated by how that coverage distracts from the more immediate problems.
We are building products on a fundamentally insecure piece of technology that multiple researchers are starting to suggest might be impossible to secure, and basically everyone in the industry is completely ignoring that and just launching their products anyway, because they've already invested too much VC money into the hype cycle and are unwilling to hit the brakes and admit there might be critical unsolved problems.
It's not so much, "look at these Rationalists, they're so silly that's not a real concern" as much as "look at these Rationalists, they're so worried about the future that they don't realize that their concerns about AGI have basically been coopted into propaganda for OpenAI to distract from the fact that GPT has fundamental security concerns that have to be solved before people can build products on top of this." The EA/Rationalist community doesn't seem to realize that their talk about paperclip maximizers is part of the reason why ordinary people don't believe security researchers when they're told just how gullible and unpredictable LLMs are. They are throwing away short-term concerns to talk about long-term concerns that are poorly understood and difficult to prepare for, and in the process they are increasing the speed of irresponsible AI development and making even their own concerns harder to address.
OpenAI loves to have conversations about AGI. It loves to do that kind of security research. It's not as fond of talking about prompt injection, because the news articles that get generated when it talks about paying humans to solve captchas are much better press coverage than the news articles talking about how reading the wrong web page can cause it to start phishing the user.
There is a kind of hijacking of EA fear that's happening here, where every opinion piece by Elieser about bombing chipset production companies actually increases people's confidence in the current products and makes the casual reader believe that LLMs are smarter and more secure and more human-like and reliable than they actually are, and increases the likelyhood that some VC firm runs out and starts wiring them into police robots.
And I think that very often the Rationalist/EA response to this kind of critique is "yes, those problems are bad, but AGI is worse. It's existential." And my response to that would be that if you're worried about AGI, you're picking arguably the worst strategy possible to try and prevent it. The demonstrable security/privacy risks around LLMs are more than enough to call for caution and pause around their deployment in real-world applications; you don't need to resort to having conversations that (fairly or not) come across to an average person as basically sci-fi storytelling. Especially not if those conversations are (arguably) making people more confident about LLMs than they otherwise would be.
On top of that, if you want to talk actual solutions to those existential concerns, solving prompt injection is absolutely a necessary pre-step towards achieving AI alignment. If you can't solve prompt injection, you can't "align" an AI by pretty much any definition of alignment. So it seems to me that those more "short-term" problems are pretty important to talk about and probably should be prioritized even if you're worried primarily about the long-term risks, because there's no plausible solution to those long-term risks that doesn't involve addressing the short-term risks as well.
We are building products on a fundamentally insecure piece of technology that multiple researchers are starting to suggest might be impossible to secure, and basically everyone in the industry is completely ignoring that and just launching their products anyway, because they've already invested too much VC money into the hype cycle and are unwilling to hit the brakes and admit there might be critical unsolved problems.
It's not so much, "look at these Rationalists, they're so silly that's not a real concern" as much as "look at these Rationalists, they're so worried about the future that they don't realize that their concerns about AGI have basically been coopted into propaganda for OpenAI to distract from the fact that GPT has fundamental security concerns that have to be solved before people can build products on top of this." The EA/Rationalist community doesn't seem to realize that their talk about paperclip maximizers is part of the reason why ordinary people don't believe security researchers when they're told just how gullible and unpredictable LLMs are. They are throwing away short-term concerns to talk about long-term concerns that are poorly understood and difficult to prepare for, and in the process they are increasing the speed of irresponsible AI development and making even their own concerns harder to address.
OpenAI loves to have conversations about AGI. It loves to do that kind of security research. It's not as fond of talking about prompt injection, because the news articles that get generated when it talks about paying humans to solve captchas are much better press coverage than the news articles talking about how reading the wrong web page can cause it to start phishing the user.
There is a kind of hijacking of EA fear that's happening here, where every opinion piece by Elieser about bombing chipset production companies actually increases people's confidence in the current products and makes the casual reader believe that LLMs are smarter and more secure and more human-like and reliable than they actually are, and increases the likelyhood that some VC firm runs out and starts wiring them into police robots.
And I think that very often the Rationalist/EA response to this kind of critique is "yes, those problems are bad, but AGI is worse. It's existential." And my response to that would be that if you're worried about AGI, you're picking arguably the worst strategy possible to try and prevent it. The demonstrable security/privacy risks around LLMs are more than enough to call for caution and pause around their deployment in real-world applications; you don't need to resort to having conversations that (fairly or not) come across to an average person as basically sci-fi storytelling. Especially not if those conversations are (arguably) making people more confident about LLMs than they otherwise would be.
On top of that, if you want to talk actual solutions to those existential concerns, solving prompt injection is absolutely a necessary pre-step towards achieving AI alignment. If you can't solve prompt injection, you can't "align" an AI by pretty much any definition of alignment. So it seems to me that those more "short-term" problems are pretty important to talk about and probably should be prioritized even if you're worried primarily about the long-term risks, because there's no plausible solution to those long-term risks that doesn't involve addressing the short-term risks as well.
This is absolutely unreal. They want it to execute battle ops. As in, kill people. Using chat prompts. What could go wrong?
Some generals believe that an army of machines is simply a more efficient army at their disposal. But what AI really enables is excluding those pesky generals, and all other humans, from the chain of command. AI is that One Ring to rule them all.
Makes me think of the Star Trek episode “A Taste of Armageddon” where a computer simulates the outcome of a war and its causalities, and then those civilizations actually go to murder that number of people in real life.
That was fast. Wondering what the pace of work is like at Palantir to have shipped this so quickly.
When I interviewed with them in 2018 they plainly stated that most teams expect 60+ hours. One of the interviewers said she works 10 hours a day, 7 days a week. She did, however, insist that it was a personal choice to do so.
I did not end up working there.
I did not end up working there.
Oh wow. That’s a no from me dawg.
Anecdotally, I spent a Thanksgiving with a family friend, besides an hour for the meal he worked all day on a project due the following day.
Killer demo...
[deleted]