Face swapping with Python, dlib, and OpenCV(matthewearl.github.io)
matthewearl.github.io
Face swapping with Python, dlib, and OpenCV
http://matthewearl.github.io/2015/07/28/switching-eds-with-python/
14 comments
https://vimeo.com/29348533
Wow! That is creepy and impressive at the same time.
See also https://www.youtube.com/watch?v=i2VvvOWWquo using dlib and opencv.
Oh wow. This will make a great nightmare.
Nice work! But people should be aware that this type of thing is patented left right and sideways. I wrote a patent for automated actor replacement in filmed media in '05, filed it globally, and it began being awarded in '08 in the various regions it was filed. Here's one of the US versions: http://www.google.com/patents/WO2007035558A2?cl=en I no longer own it; now it is part of a broad family of patents collectively forming a "Personalized Media" infrastructure. It is owned by a major media company I'm not allowed to disclose.
Essentially, it is possible to replace people, objects, environments, & audio elements automatically through a series of techniques. Those techniques form the basis of a powerful advertising and message delivery infrastructure. And the collective IP is already owned by a major media conglomerate.
Essentially, it is possible to replace people, objects, environments, & audio elements automatically through a series of techniques. Those techniques form the basis of a powerful advertising and message delivery infrastructure. And the collective IP is already owned by a major media conglomerate.
This is literally what's wrong with the patent system.
Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.
I bet there are a million patents out there for 'automated face replacement'. I just hope very very much, that somewhere there is some prior art that makes them all completely worthless and bankrupts whomever is making any money from them.
Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.
I bet there are a million patents out there for 'automated face replacement'. I just hope very very much, that somewhere there is some prior art that makes them all completely worthless and bankrupts whomever is making any money from them.
What you said was a little strong, but I do fundamentally agree with you that patent clerks do not perhaps have the benefit of 5-year technology trajectory to make informed decisions.
When you look at the patents that Newegg fights (and wins), clearly something has gone wrong at USPTO.
When you look at the patents that Newegg fights (and wins), clearly something has gone wrong at USPTO.
face is just a particular pattern, pattern matching and replacing was all done since the sixties.
Patents are for particular methods of doing this. You can't reduce all patents to "moving atoms and molecules around in different configurations" claim prior art, and invalidate all past, present and future patents.
To illustrate this, a patent (as originally intended at least) wouldn't be for something broad like "walking down the street" but maybe for something more like "walking down the street with a particular gait."
While I don't necessarily agree with the patent system, it detracts from the conversation with it's not an informed discussion.
To illustrate this, a patent (as originally intended at least) wouldn't be for something broad like "walking down the street" but maybe for something more like "walking down the street with a particular gait."
While I don't necessarily agree with the patent system, it detracts from the conversation with it's not an informed discussion.
True that the face is just a pattern, but matching and replacing in a manner that a human viewer is not aware of the manipulation is an entirely different matter. Integrated lighting, media grain, obscuring elements, including any reflections of your subject in the replacement, matching motion blur aliasing and so forth form the realistic problem to solve.
The automated face replacement patents begin in the late 70's. By '05 the process had to be entire in-motion actor replacement to be patentable. To some degree, I'd say patents force people to work in uncharted areas.
You won't believe this one weird trick to save money on a network of local nightly news stations!
>> Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.
Who's discouraging? Being aware that it's highly patented doesn't mean you shouldn't play around with it, it just means that you probably shouldn't be trying to make money off of someone else's work.
Who's discouraging? Being aware that it's highly patented doesn't mean you shouldn't play around with it, it just means that you probably shouldn't be trying to make money off of someone else's work.
The thing with software patents though is that most of the time they've preventing you from making money off of your own work, just because someone at one point did that same work first. Patents are needed in industries where r&d is expensive enough to warrant protecting inventors for a sizeable time so they can recoup their investment. Software r&d is too cheap for patents on software to do anything but harm.
As a quick rule of thumb, whenever people are being advised never to look at patents, as they are in software, that's when those patents do more harm than good.
As a quick rule of thumb, whenever people are being advised never to look at patents, as they are in software, that's when those patents do more harm than good.
I agree with the first part, and disagree with the second part of your statement.
Just because there are patents doesn't mean a person can't play around and even try to make money off of their work. It just means they need to do a little bit of investigation before trying to monetize their work.
Just because there are patents doesn't mean a person can't play around and even try to make money off of their work. It just means they need to do a little bit of investigation before trying to monetize their work.
It's not clear that the linked patent applies to the technique in the original post. https://www.google.com/patents/US7787664 and https://www.google.com/patents/US8712189 both seem more relevant.
Just because something looks like it could be patent encumbered doesn't mean you shouldn't pursue it. Patents need to be fought over in court. It costs millions of dollars to bring a patent lawsuit. In the event that your little side project or startup becomes a target for a patent lawsuit, frivolous or otherwise, you've likely already won.
Even with a semi-broken patent system, the world simply favors active, high-quality execution over passive "intellectual property" generation. Focus 364 days a year on a building a great product that people love and then spend a day to talk to your attorney about patents, IP, and how to protect the valuable company that you've built by focusing on your customers.
Jeff Schox has a great talk on patents and startups that is very relevant to this discussion: https://www.youtube.com/watch?v=hyv75d4JG7c
Just because something looks like it could be patent encumbered doesn't mean you shouldn't pursue it. Patents need to be fought over in court. It costs millions of dollars to bring a patent lawsuit. In the event that your little side project or startup becomes a target for a patent lawsuit, frivolous or otherwise, you've likely already won.
Even with a semi-broken patent system, the world simply favors active, high-quality execution over passive "intellectual property" generation. Focus 364 days a year on a building a great product that people love and then spend a day to talk to your attorney about patents, IP, and how to protect the valuable company that you've built by focusing on your customers.
Jeff Schox has a great talk on patents and startups that is very relevant to this discussion: https://www.youtube.com/watch?v=hyv75d4JG7c
Be aware that a single technique can infringe on multiple patents, and a patent can extend other patent(s), triggering a situation where a single technique requirements licensing from multiple entities.
It is exactly the issue of fighting patents costing millions of dollars that makes technology development financially risky. When accused of infringing, one has to respond, and that response costs both significant time and significant money.
I agree a side project need not worry. A startup does. I also agree that the world favors active execution over passive intellectual property generation. Our world also forces the active executing organization to protect itself if they need any financing to realize their goals.
It is exactly the issue of fighting patents costing millions of dollars that makes technology development financially risky. When accused of infringing, one has to respond, and that response costs both significant time and significant money.
I agree a side project need not worry. A startup does. I also agree that the world favors active execution over passive intellectual property generation. Our world also forces the active executing organization to protect itself if they need any financing to realize their goals.
> Company I am not allowed to disclose.
Aren't patents supposed to be public information ? One of the (rather hollow in practice) justification for patents is to (also) encourage eventual disclosure of knowledge.
Aren't patents supposed to be public information ? One of the (rather hollow in practice) justification for patents is to (also) encourage eventual disclosure of knowledge.
> Aren't patents supposed to be public information ?
Private agreements can restrict a person's comment on things without regard to whether the information is already public.
Private agreements can restrict a person's comment on things without regard to whether the information is already public.
The patents are, but their ownership does not have to be.
> The patents are, but their ownership does not have to be.
I disagree.
It's important that people be able to determine whether the patent is in use or held by an NPE that's obligated to license it to people who want to implement products.
Were I to want to work in motion capture, I have no way to ascertain who owns your patents, if they're an NPE, and what possible licensing options exist.
I think patent owners should be forced to register with the patent office in a look-up table if they want to hold their patents.
I disagree.
It's important that people be able to determine whether the patent is in use or held by an NPE that's obligated to license it to people who want to implement products.
Were I to want to work in motion capture, I have no way to ascertain who owns your patents, if they're an NPE, and what possible licensing options exist.
I think patent owners should be forced to register with the patent office in a look-up table if they want to hold their patents.
I think the GP was just stating the fact, not expressing his opinion about it. But I agree that secret patent ownership sounds a little dodgy. Say I want to apply for a license to use some patented technology? Who am I supposed to turn to?
[deleted]
True. But in the case where the owner is secret, they shouldn't be able to sue infringers (I wish).
... and people should also be aware those patents might be invalidated by re-examination or an action in federal court. I don't know why you aren't allowed to disclose who now 'owns' the patent you refer to. It is trivial to locate this information with the link you provide and some googling.
Also your statement "the collective IP is already owned by a major media conglomerate" if taken to refer to totality of technology which makes it "possible to replace people, objects, environments, & audio elements automatically through a series of techniques" is false.
Also your statement "the collective IP is already owned by a major media conglomerate" if taken to refer to totality of technology which makes it "possible to replace people, objects, environments, & audio elements automatically through a series of techniques" is false.
[deleted]
I beg to differ, there most definitely is a series of patented technologies that in total enable replacement of people, objects, environments and audio elements in traditional media. That key is traditional, linear media. New forms of media, interactive forms, VR/AR and hybrids are wide open fields. That's a big reason new fields of technology are so exiting - it's not just interesting to us geeks, but valuable to the businesspeople.
Only until it's locked down and killed for 20 years.
Even if it's patented you can still play around with the technology yourself. Just don't sell it or make a product out of it.
> It is owned by a major media company I'm not allowed to disclose.
I don't understand this. If the patents are public, why shouldn't the owners be as well?
I don't understand this. If the patents are public, why shouldn't the owners be as well?
As far as I can tell, this is just a patent troll's patent. It seems to be a very generic process for replacing people, objects, environments, etc.
It's a patent and you're not allowed to disclose its owner? Is that a thing? Secret patents?
Those were the terms of the sale. I was not in a position to argue.
And it prohibits disclosure even after the sale/reassignment was completed? I don't understand the purpose of that, at least not a legitimate purpose.
http://assignment.uspto.gov/#/search?q=7460731
Here is the most recent public assignment of that patent family. It from Flixor, Inc. to Big Tent Entertainment, LLC.
http://www.bigtent.tv/
Sounds like you were on the losing end of some strong arm tactics. Or did I read your comment wrongly?
What's your view on the patent? Pretty broad? Or a sign of very quick progress that in 10 yrs something innovative (hence, a patent) turns into a few open source libraries and few lines of code to replicate. Most responses (including my thought) call 'patent troll' etc, but perhaps there was substantial thought and effort in the techniques below the patent.
The majority of these "Personalized Media" patents rely on existing integrated media - meaning imagery and audio composed of multiple elements - and the ability to isolate one element, remove or modify it, and/or replace it with something else in a manner that the original media's intent and message is retained, only now it is enhanced for a specific audience.
Think product placements that change for every single viewer.
Real time rendering, VR/AR and that entire enchilada of media techniques completely sidestep these patents because there is no original media, it's all created and composited on the fly, dynamically for the single specific consumer of the media.
Think product placements that change for every single viewer.
Real time rendering, VR/AR and that entire enchilada of media techniques completely sidestep these patents because there is no original media, it's all created and composited on the fly, dynamically for the single specific consumer of the media.
Just to echo this a bit, I spent 3 years working on image similarity tech and a lot of our research was challenging, not because of the intrinsic complexity of the problem, but because we had to implement it in such a way as to step around existing patents.
I guess you found the math combined with the search for alternative routes interesting? But did you also try to find prior art? My first thought would be that for unreasonably broad patents prior art is the easiest way to be able to use the tech?
I wouldn't say the patents we hit were broad. In some cases we thought they were fairly obvious, but that's much more subjective. We didn't chase prior art, mainly because none of the barriers were unavoidable and we wanted to keep momentum on the build.
I only had one foot in the algo camp to be honest. My main task was making the query engine scale.
I only had one foot in the algo camp to be honest. My main task was making the query engine scale.
Bleurgh.
Would you rather people spend time developing in this area only to learn they can't use their work?
No, he would rather be free to use the code he wrote.
Which is the exact situation I found myself in: I began developing in this area, and learned of patents I needed to sidestep. Realizing a hole in the patent landscape, I filed to protect what I was doing.
So now other people have to sidestep a sidestep?
It's only an issue because the original "sidestepper" sold-off, or was perhaps bullied into selling (theory), the patent to an entity. An entity that will unfortunately just hold it in a "bag of tricks" until they need it, or decide to play with it, or decide to litigate with, or decide to "license" out usage of it to another big player willing to fork over money for it.
Had an innovative tinkerer kept it, then who knows what might have happened. But I'd take a wild guess and say that the odds are more likely that a such an innovator would allow others the use/play/profit off of their novel ideas than would a large entity that sees it as a potential avenue of competition.
Had an innovative tinkerer kept it, then who knows what might have happened. But I'd take a wild guess and say that the odds are more likely that a such an innovator would allow others the use/play/profit off of their novel ideas than would a large entity that sees it as a potential avenue of competition.
I'm not saying that. It's just frustrating.
Try doing this 10 years ago, putting your life savings into it, and then VCs and media companies "not getting it", your money runs out, and then you have to sell your IP simply to pay the rent.
Way to play the victim card, after first playing the everyone-else-was-doing-it card!
"Los Angeles based MBA..."
Makes sense, I guess you learnt this two card trick in MBA 101 :)
"Los Angeles based MBA..."
Makes sense, I guess you learnt this two card trick in MBA 101 :)
I'm not meaning to criticize you personally (at all). I'm just saying that the idea of being able to "declare ownership" of what is essentially a series of mathematical operations just doesn't feel quite right.
I realise that people have to make money some how, but developing a useful, competitive, monetisable product that people are able to choose to buy is, in my opinion, the way forwards. Patenting the technology so that you can operate without competition rather than developing your product to stay ahead of the competition "organically" seems like the wrong way of doing it to me.
Perhaps I'm an idealist, and need to get real...
I realise that people have to make money some how, but developing a useful, competitive, monetisable product that people are able to choose to buy is, in my opinion, the way forwards. Patenting the technology so that you can operate without competition rather than developing your product to stay ahead of the competition "organically" seems like the wrong way of doing it to me.
Perhaps I'm an idealist, and need to get real...
I agree with you. But the patent system exists... A startup has to protect its work. Plus, when pitching for financing the first set of questions include "Is this patented?" If you answer no and don't have a valid reason why not, they tell you to get it patented and come back when that is done.
So, there are an infinite number of things that are just series of mathematical operations. It's the ones that are useful that matter. Why would we not compensate people for discovering them?
This is a good comment, much better than the "Bleurgh" one, so I transferred you a vote (down there, up here).
If you and people like you hadn't patented these things, they would've been able to use their work.
Thanks for the warning, at least.
:(
Thanks for the warning, at least.
:(
> If you and people like you hadn't patented these things, they would've been able to use their work.
Someone else would have done it and you would be in the same position you are now.
Someone else would have done it and you would be in the same position you are now.
The patent system exists. When I filed, there were already dozens of patents in this area. I was simply trying to protect my own work for my startup.
Your argument seems to be "everyone else is doing it, so it's fine". I'm not having a go at you, but that doesn't seem like a very strong argument.
I needed financing for my startup based on this technology. After an initial financing search, it became clear it had to be patented to get financing.
To some degree, patents are a mechanism forcing early technology companies into expensive legal engagements, which drain resources from the company, triggering hefty financing requirements, and a dilution of the inventor's equity.
It is a carefully crafted system that benefits the 1% entirely, and the remaining 99% are merely the crop they farm.
To some degree, patents are a mechanism forcing early technology companies into expensive legal engagements, which drain resources from the company, triggering hefty financing requirements, and a dilution of the inventor's equity.
It is a carefully crafted system that benefits the 1% entirely, and the remaining 99% are merely the crop they farm.
Don't worry about the naysayers. I am in the same position as you right now as we are doing work with AR and computer vision that people/investors etc... don't get yet. Maybe we are too early in the market, but probably not. If we are too early then at least we have something that we can fall back on without losing everything we have put into it.
At the end of the day I have to be able to say that I did everything I could to make our technology "win" in the marketplace.
At the end of the day I have to be able to say that I did everything I could to make our technology "win" in the marketplace.
If we are too early then at least we have something that we can fall back on without losing everything we have put into it.
No, see, what'll happen is your patent will be bought up during your exit event/firesale, and then used to harass other people in the space.
Especially in AR/VR/whatever, there's a huge incentive to patent things which are fairly obvious with just a little applied engineering time, and which will be used to harass a lot of people later.
If you wanted your technology to "win", you should make it open and free (in the software sense) to use--as it is, you've just got some tepid AR entry in the backwards swamp that is AEC.
No, see, what'll happen is your patent will be bought up during your exit event/firesale, and then used to harass other people in the space.
Especially in AR/VR/whatever, there's a huge incentive to patent things which are fairly obvious with just a little applied engineering time, and which will be used to harass a lot of people later.
If you wanted your technology to "win", you should make it open and free (in the software sense) to use--as it is, you've just got some tepid AR entry in the backwards swamp that is AEC.
Ha thanks for all the kind words. Actually our patents don't have to do with AEC and are actually pretty novel in the computer vision realm.
I actually ran the idea of putting our work for "free" by our current investors because I am a fan of FOSS but it's a non-starter when it comes to raising money and it's just a fact of life that we need investment to keep going.
I actually ran the idea of putting our work for "free" by our current investors because I am a fan of FOSS but it's a non-starter when it comes to raising money and it's just a fact of life that we need investment to keep going.
At the end of the day, business is business. Good luck to you and your team!
I'm curious, did you put together an algorithm and\or program which actually did this automated actor replacement or did you just patent the idea? No judgement either way, "don't hate the player..." and all that :)
edit: looking at your bio it seems pretty likely that you had properly implemented it!
edit: looking at your bio it seems pretty likely that you had properly implemented it!
Fully implemented, but expensive to use. It's modern visual effects technology, automated. Also, not a single algorithm or program, but a media pipeline of over a dozen custom programs, plus integration with the major VFX software packages.
> It is owned by a major media company I'm not allowed to disclose.
Why would a patent owner want to remain unknown?
Why would a patent owner want to remain unknown?
Come on guys. Like all patents, the owner must be disclosed to the USPTO. A quick Google search turns up Flixor as the original owner, and "Big Tent Entertainment, LLC" as the current owner of some of the recent patents in this family. A further search of their website indicates it licenses various media technology to companies including "Concept One, Nintendo, Fifth Sun, Parragon Publishing, Harry Abrams, Kids Station, Merch Source and more".
That's one reason for the question! Why would a patent owner want to prevent someone from divulging public information.
Another question being is the owner of record merely a front, and for what purpose?
Another question being is the owner of record merely a front, and for what purpose?
Likely, this is a standard term in their inventor assignment contract. When bsenftner signed it, the patent applications were probably not public yet (there is an 18 month period between filing and publication in which the application remains "secret"), and during that period it makes sense to have a non-disclosure term.
As for the current owner being a front... if it is, it's not a very good one (real patent troll companies are much better at it, using shell corporations in various jurisdictions, each tied to a single or a small number of patent families).
As for the current owner being a front... if it is, it's not a very good one (real patent troll companies are much better at it, using shell corporations in various jurisdictions, each tied to a single or a small number of patent families).
Not true. The applicant must be disclosed during the application process, but the owner can be legally change without notifying the PTO or updating ownership records. Assignments can be registered, but you don't have to register assignments. So Big Tent could theoretically sell the patents to another party and never register the sale with the PTO.
For a patent troll it makes perfect sense. Rig the line and wait for a bite.
trolling
Noob question: What if the developer who "might" have infringed on the patent, open sources his work, and it's made available on github with multiple forks, and now there's a whole community behind it, and so on?
[deleted]
Certain specific processes may be patented, but "this type of thing" cannot be patented outright. Ideas are not patentable, processes are.
[deleted](1)
OpenCV is an awesome Swiss-army steam-shovel (thank you Willow Garage!)
In the 90's, there was Kai's Power Goo (now owned by Corel) and probably something similar used by Conan O'Brien's art dept. to produce the "If They Mated" segments. It's such a shame tools owned by big companies are effectively locked away from ever being marketed again by asserting IP rights.
I really think intellectual property needs a principle similar to adverse possession [0]; that is, if it's not being actively used in a meaningful, sustained venture, another shop can reboot it. (Giving capitalism the right to fork.) Btw, I'm not the first, nor the last, to conceive of this. [1]
0. https://en.wikipedia.org/wiki/Adverse_possession
1. http://jolt.law.harvard.edu/articles/pdf/v16/16HarvJLTech327...
In the 90's, there was Kai's Power Goo (now owned by Corel) and probably something similar used by Conan O'Brien's art dept. to produce the "If They Mated" segments. It's such a shame tools owned by big companies are effectively locked away from ever being marketed again by asserting IP rights.
I really think intellectual property needs a principle similar to adverse possession [0]; that is, if it's not being actively used in a meaningful, sustained venture, another shop can reboot it. (Giving capitalism the right to fork.) Btw, I'm not the first, nor the last, to conceive of this. [1]
0. https://en.wikipedia.org/wiki/Adverse_possession
1. http://jolt.law.harvard.edu/articles/pdf/v16/16HarvJLTech327...
And it had to be fucking Miliband?!
Well that said: https://www.reddit.com/r/gentlemangabers
(Before you go dig up the actual source code: it just uses a web service to get the coordinates of the faces)
Well that said: https://www.reddit.com/r/gentlemangabers
(Before you go dig up the actual source code: it just uses a web service to get the coordinates of the faces)
>And it had to be fucking Miliband?!
Did you see what happened here the last time someone used Lena in their example?
Did you see what happened here the last time someone used Lena in their example?
No, I did not.
By the way, I was shocked the first time I saw what the full picture looked like... 8-)
By the way, I was shocked the first time I saw what the full picture looked like... 8-)
A lot of the discussion was focused on the use of a crop of the Lena image. https://news.ycombinator.com/item?id=8704629
>I was shocked the first time I saw what the full picture looked like.
Me too! Ironically I probably wouldn't have ever seen the uncropped image if not for the controversy around the common use of the cropped version.
>I was shocked the first time I saw what the full picture looked like.
Me too! Ironically I probably wouldn't have ever seen the uncropped image if not for the controversy around the common use of the cropped version.
Haha, my God. Kids.
This is exactly the kind of thing that Menpo makes easy:
https://gist.github.com/jabooth/a52019f6e7c09f2a3e01
(Forgive the rushed nature of this, literally just whipped this up in the last 20 mins. Will provide a link to the menpofit model in a later revision)
(Forgive the rushed nature of this, literally just whipped this up in the last 20 mins. Will provide a link to the menpofit model in a later revision)
Matching the low frequency image information is a good simple approach to do the blending, but better results can be achieved with Poisson image editing:
http://www.cs.jhu.edu/~misha/Fall07/Papers/Perez03.pdf
The amount of mathematics that goes into automating faceswaps is awesome. The fact that anyone can do this at home with the sole purpose of generating a funny image is great!
How long does this take to run? Could it be done in real-time?
This was a great and interesting article to read. I have really enjoyed all of this very cool information. <a href="http://www.thecasesolutions.com/hozho-a-16335">Hozho Case Solution</a>
Isn't this like totally giving away the Enquirer's[1] secrets? I enjoyed the discussion too. Next up, deriving a set of makeup (cosmetics) steps that need to happen to have your face recognized as some other persons face.
[1] a gossip newspaper which often (always?) has dubious photographic evidence backing up their claims.
[1] a gossip newspaper which often (always?) has dubious photographic evidence backing up their claims.
Neil deGrasse Tyson with Peter Serafinowicz face: https://youtu.be/gr4gIK7Bv3E
Had fun over the weekend playing with this system. :)
Had fun over the weekend playing with this system. :)
The result doesn't look like Ed Milliband, though...
Well, it looks something between Milliband and Voldemort...
This is awesome. How long did it take to develop?