Facebook open-sources deep-learning modules(research.facebook.com)
research.facebook.com
Facebook open-sources deep-learning modules
https://research.facebook.com/blog/879898285375829/fair-open-sources-deep-l20earning-modules-for-torch/
75 comments
Here's a link to the patent clause: https://github.com/facebook/fbcunn/blob/master/PATENTS
DannyBee, I'm having a hard time envisioning what sort of scenario might trigger this, and what the repercussions might be. Any chance you could explain?
DannyBee, I'm having a hard time envisioning what sort of scenario might trigger this, and what the repercussions might be. Any chance you could explain?
Do you mean the first part, or the validity provision?
as a reply points out, it is incredibly broad.
Assuming you mean the first part (the validity provision seems "obviously bad" to me), let's lay out some scenarios:
You make your important software dependent on this software. Congrats, facebook can now sue you with impunity because you can't afford to lose these patents.
Worse, the fact that they have given you a grant does not prevent them from suing you, and the second you counterclaim, you lose. Note also that the grant specifically does not cover "the Software in combination with any software or other technology provided by you or a third party." So if you take this stuff and run it in a pipeline, and facebook has a patent on running it in a pipeline, you don't get that patent and they can still sue you.
(Again, in the apache case, you'd only ever lose the patent grant if you sued facebook over the software in question).
Ignoring direct cases, facebook can also use this as leverage during acquisitions that use their software (Nice software, shame if you were to lose rights to it), etc.
There are potentially an infinite number of ways to use this kind of stuff.
Also note this kind of thing has been looked at in the past. There is an entire discussion here: http://mail-archives.apache.org/mod_mbox/archive-license/200...
from when v2 of apache license was being written, etc.
There are reasons the Apache grant has become pretty much the standard - most people and corporations find it a pretty good balance of various concerns (being able to defend yourself in patent lawsuits, etc).
What Facebook has done is entirely one-sided. It benefits Facebook only, and even then, it generally only helps them in situations where they want to sue people. None of this really helps defensively. If they wanted it for defense, they'd write it differently to cover defensive situations.
Assuming you mean the first part (the validity provision seems "obviously bad" to me), let's lay out some scenarios:
You make your important software dependent on this software. Congrats, facebook can now sue you with impunity because you can't afford to lose these patents.
Worse, the fact that they have given you a grant does not prevent them from suing you, and the second you counterclaim, you lose. Note also that the grant specifically does not cover "the Software in combination with any software or other technology provided by you or a third party." So if you take this stuff and run it in a pipeline, and facebook has a patent on running it in a pipeline, you don't get that patent and they can still sue you.
(Again, in the apache case, you'd only ever lose the patent grant if you sued facebook over the software in question).
Ignoring direct cases, facebook can also use this as leverage during acquisitions that use their software (Nice software, shame if you were to lose rights to it), etc.
There are potentially an infinite number of ways to use this kind of stuff.
Also note this kind of thing has been looked at in the past. There is an entire discussion here: http://mail-archives.apache.org/mod_mbox/archive-license/200...
from when v2 of apache license was being written, etc.
There are reasons the Apache grant has become pretty much the standard - most people and corporations find it a pretty good balance of various concerns (being able to defend yourself in patent lawsuits, etc).
What Facebook has done is entirely one-sided. It benefits Facebook only, and even then, it generally only helps them in situations where they want to sue people. None of this really helps defensively. If they wanted it for defense, they'd write it differently to cover defensive situations.
> You make your important software dependent on this software. Congrats, facebook can now sue you with impunity because you can't afford to lose these patents.
I have been rattling the clause jawns posted around in my head, and I cannot see this. I agree that the clause is onerous and I would not incorporate software that had such a clause into one of my products without a backup plan. I do not see how this gives Facebook the ability to sue anyone with impunity. The clause applies to alleging patent infringement BY Facebook or its subsidiaries (i.e., you are suing them) or if Facebook is specifically suing you for patent infringement. Clause section b only applies if your defense is based on invalidating a Facebook patent; saying, "We don't actually infringe that" does not trigger this clause.
I have been rattling the clause jawns posted around in my head, and I cannot see this. I agree that the clause is onerous and I would not incorporate software that had such a clause into one of my products without a backup plan. I do not see how this gives Facebook the ability to sue anyone with impunity. The clause applies to alleging patent infringement BY Facebook or its subsidiaries (i.e., you are suing them) or if Facebook is specifically suing you for patent infringement. Clause section b only applies if your defense is based on invalidating a Facebook patent; saying, "We don't actually infringe that" does not trigger this clause.
First, The only thing most companies have they could use against facebook, if facebook were to sue them, are patents.
Second, "saying, "We don't actually infringe that" does not trigger this clause."
This is false. Such a claim is a claim that facebook's patent claim is unenforceable against you. This falls directly into the "unenforceable" language.
Also note that you claim it requires lawsuits. Actually, as worded, it does not.
In fact, it says " The license granted hereunder will terminate, automatically and without notice, for anyone that makes any claim (including by filing any lawsuit, assertion or other action) alleging". It specifically says "assertion or other action". It does not require lawsuits.
So arguably, just by posting a blog post that says facebook's patent is invalid, you'd lose rights, because that is an assertion that a right in a patent of facebook's is invalid.
If it's not meant to cover this, it's not at all clear what the difference between "an assertion" and "a lawsuit" are supposed to be in the examples, or what "other action" would constitute.
Second, "saying, "We don't actually infringe that" does not trigger this clause."
This is false. Such a claim is a claim that facebook's patent claim is unenforceable against you. This falls directly into the "unenforceable" language.
Also note that you claim it requires lawsuits. Actually, as worded, it does not.
In fact, it says " The license granted hereunder will terminate, automatically and without notice, for anyone that makes any claim (including by filing any lawsuit, assertion or other action) alleging". It specifically says "assertion or other action". It does not require lawsuits.
So arguably, just by posting a blog post that says facebook's patent is invalid, you'd lose rights, because that is an assertion that a right in a patent of facebook's is invalid.
If it's not meant to cover this, it's not at all clear what the difference between "an assertion" and "a lawsuit" are supposed to be in the examples, or what "other action" would constitute.
> First, The only thing most companies have they could use against facebook, if facebook were to sue them, are patents.
Facebook sues me for stealing proprietary information after hiring one of their engineers. I have no patents. How do patents help me?
> This is false. Such a claim is a claim that facebook's patent claim is unenforceable against you. This falls directly into the "unenforceable" language.
It is not false. Claiming a patent is unenforceable is not the same as claiming something you are doing in a specific case is not covered by a patent. "Unenforceable" has a specific legal definition.[1] The list of common actions that argue unenforceability: patent is invalid; inequitable conduct by patentee; delay in braining suit for infringement; patent misuse. Please explain how "our product doesn't infringe this patent" meets this definition.
> Also note that you claim it requires lawsuits. Actually, as worded, it does not.
I did not state that it requires lawsuits. You claimed the clause grants Facebook the ability to sue with impunity, and I was using the clauses to refute that statement in that context. I am well aware that the clause, as written, does not require lawsuits.
[1] http://www.legalmatch.com/law-library/article/patent-infring...
Facebook sues me for stealing proprietary information after hiring one of their engineers. I have no patents. How do patents help me?
> This is false. Such a claim is a claim that facebook's patent claim is unenforceable against you. This falls directly into the "unenforceable" language.
It is not false. Claiming a patent is unenforceable is not the same as claiming something you are doing in a specific case is not covered by a patent. "Unenforceable" has a specific legal definition.[1] The list of common actions that argue unenforceability: patent is invalid; inequitable conduct by patentee; delay in braining suit for infringement; patent misuse. Please explain how "our product doesn't infringe this patent" meets this definition.
> Also note that you claim it requires lawsuits. Actually, as worded, it does not.
I did not state that it requires lawsuits. You claimed the clause grants Facebook the ability to sue with impunity, and I was using the clauses to refute that statement in that context. I am well aware that the clause, as written, does not require lawsuits.
[1] http://www.legalmatch.com/law-library/article/patent-infring...
Presumably their "assertion or other action" could cover things like the patents stack exchange too? If they make one of those common terrible patents, and you point out prior art somewhere public like that, you'd seem to lose all these licenses.
How can they claim this text file applies?
There is also a file with the industry-standard name LICENSE (https://github.com/facebook/fbcunn/blob/master/LICENSE), with a standard looking BSD license in it.
Surely somebody using it could claim that was the file that applied, and this other non-standard whimsical "PATENTS" file was irrelevant?
Or can people just bury random licenses deep inside copyrighted works, and expect them to apply?
There is also a file with the industry-standard name LICENSE (https://github.com/facebook/fbcunn/blob/master/LICENSE), with a standard looking BSD license in it.
Surely somebody using it could claim that was the file that applied, and this other non-standard whimsical "PATENTS" file was irrelevant?
Or can people just bury random licenses deep inside copyrighted works, and expect them to apply?
Does that BSD license grant a patent license though? It doesn't seem to. Otherwise arguing their "PATENTS" file doesn't apply means you have no right to any patents exercised by the software anyway.
All it does is grant rights, albeit flimsy ones that can be taken away in stupid circumstances. There's no real use arguing it doesn't count because then you're just saying you never had those right and are infringing in exactly the same way.
All it does is grant rights, albeit flimsy ones that can be taken away in stupid circumstances. There's no real use arguing it doesn't count because then you're just saying you never had those right and are infringing in exactly the same way.
> The license granted hereunder will terminate, automatically and without notice, for anyone that makes any claim ... by ... assertion or other action ... alleging .. that any right in any patent claim of Facebook is invalid or unenforceable.
That sounds pretty chilling considering that could mean that anything any of your employees say about any other Facebook patent could wreck your business, if you choose to use Facebook code from Github.
That sounds pretty chilling considering that could mean that anything any of your employees say about any other Facebook patent could wreck your business, if you choose to use Facebook code from Github.
The most interesting part to me is whether or not
such claim is related to the Software - so any patent suit against any software, product or
service of Facebook or any of its subsidiaries or affiliates terminates patent license to this software... seems like they went for the nuclear defense!
Does this apply in continental Europe as well? I doubt so. You can't give away your rights via any form of contract.
As long as you're doing your training offline and shipping the resulting learned models, I don't see this is a big risk. If you're planning on shipping software that can learn online, then sure, use something else for that, or ask for a better deal from FB.
I wonder what would happen if every company offered patented algorithms under these terms. Still wouldn't stop patent trolls, I'd guess.
I wonder what would happen if every company offered patented algorithms under these terms. Still wouldn't stop patent trolls, I'd guess.
As I read it is basically says you can use Facebook's patents unless you turn around and sue them in which case the deal's off. I'm not sure it's that awful - I mean how many companies are planing to use Facebook's patents and then sue them?
You should read it again :)
It requires no lawsuits to lose patents
As written, if you make blog posts saying their patents are crap, you've lost.
(and the other provisions are not defensive, they don't require you sue facebook to lose patents)
(and the other provisions are not defensive, they don't require you sue facebook to lose patents)
This does look terrible but I'm not sure about number 2 - can you quote the part of the clause that suggested that?
"The license granted hereunder will terminate, automatically and without notice,
for anyone that makes any claim (including by filing any lawsuit, assertion or
other action) alleging ... (b) that any right in any patent claim of Facebook is invalid or
unenforceable."
(the entire rest of the ... is clause (a))
(the entire rest of the ... is clause (a))
Oops, not sure how I misread that, it's pretty clear! This unfortunately seems to reduce these projects in usefulness to pretty much awesome and interesting research - completely unusable in any business though. I don't see how any business can be expected to make all employees keep to these conditions - way too fragile and easy for an employee to screw the company.
I understand the concern. For reference, what are the worst patents that Facebook owns? Do they own patents that should not have been granted?
[deleted]
I appreciate that Facebook open-sources their libraries, but they have an awful habit of dumping source code on Github and then continuing to develop it internally without pushing those changes back out.
For example, Facebook open-sourced Thrift, then proceeded to develop it internally, and recently released out a fork of thrift, not merging anything back into the original tree.
Likewise, they open-sourced Corona a few years ago, their Hadoop fork, and I'm pretty sure that branch has been abandoned too.
So I'm really hesitant to start using any projects out of Facebook unless they develop some processes to actually maintain these projects.
For example, Facebook open-sourced Thrift, then proceeded to develop it internally, and recently released out a fork of thrift, not merging anything back into the original tree.
Likewise, they open-sourced Corona a few years ago, their Hadoop fork, and I'm pretty sure that branch has been abandoned too.
So I'm really hesitant to start using any projects out of Facebook unless they develop some processes to actually maintain these projects.
Agree that their stewardship has been lacking (especially in comparison to say Apache, Docker, etc), but for research code this is a pretty nice release in terms of the scope of problems solved. For research it's more important to have an implementation of what worked (and what /didn't/) than a polished solution + well-kept community because a lot of the current code will probably get thrown away eventually anyways. The service here is research done rather than software provided.
They've done a good job keeping up PlanOut.
https://github.com/facebook/planout
https://github.com/facebook/planout
From the article:
> We hope that these high-quality code releases will be a catalyst to the research community and we will continue to update them from time to time.
> We hope that these high-quality code releases will be a catalyst to the research community and we will continue to update them from time to time.
[deleted]
That was years ago. Things have changed around there: https://github.com/facebook
Too bad it is Torch. Working in the lua environment is not enjoyable at all. Every error becomes a long procedure of looking deep into the source of the framework since there is no type information or stack traces to go off whatsoever. You are constantly guessing about the shape of the data or what parameters are supposed to look like.
try http://deeplearning4j.org/
they just partnered with canonical: http://insights.ubuntu.com/?p=9373
they just partnered with canonical: http://insights.ubuntu.com/?p=9373
This is a recruiting move. Facebook is building its DL team, and to do that, they need to grow the community using their tools.
Paper with proper summary: http://arxiv.org/abs/1412.7580
This paper describes one part of the fbcunn release (the fast convolution layers implemented via FFT, with the source available at https://github.com/facebook/fbcunn/tree/master/src/cuda/fft). There's a lot more in fbcunn if you want to check it out.
In my experience, the fully connected layers are the bottleneck. The other issue was the alternating compute-heavy convolution and the IO-heavy pooling. I'm curious how this FFT implementation stacks up against cuDNN (what's the speedup like for just the convolutional layers? and then what's the overall speedup like?).
http://arxiv.org/pdf/1412.7580v2.pdf compares the convolutional implementation with the cuDNN layers. For the FC layers, it's just CuBLAS `sgemm`.
We've been using Torch for a while at work, really appreciate Facebook doing this. Some of us had a hard time adjusting to Lua, but it was well worth it.
I'm curious what your team experience was with Lua.
IIRC, Lua is even more liberal than Python in that if you mistype a variable name, it will silently give you 'nil' rather than throwing an exception.
Also, the prototype system is a little wonky. It seems overly flexible and makes you do too much work yourself, which would make it hard on a large team all getting started at once.
If Torch gets popular I wonder if something like TypeScript or Facebook's own Flow would start to make sense.
IIRC, Lua is even more liberal than Python in that if you mistype a variable name, it will silently give you 'nil' rather than throwing an exception.
Also, the prototype system is a little wonky. It seems overly flexible and makes you do too much work yourself, which would make it hard on a large team all getting started at once.
If Torch gets popular I wonder if something like TypeScript or Facebook's own Flow would start to make sense.
> IIRC, Lua is even more liberal than Python in that if you mistype a variable name, it will silently give you 'nil' rather than throwing an exception.
Like most things in Lua, that is easy to customize. You can just add an __index metamethod to environment tables. There are various 'strict' modules which do just that.
Like most things in Lua, that is easy to customize. You can just add an __index metamethod to environment tables. There are various 'strict' modules which do just that.
Thats what we love about Lua, it gives us control of the side effects.
So some of our team were a bit disoriented with the lack of reference counting and for others it was their first foray into functional programming.
Our team first picked up the Little Schemer and then jumped directly to lua.
Our team first picked up the Little Schemer and then jumped directly to lua.
Could you provide some steps how you could bring everyone one board? We are still having python/R migration issues.
Most of our people came from a stats background using R or python (more focused on Scipy/numpy than any other package).
A few of us had some experience with Lua before with game programming, so we took it on ourselves to teach the others Scheme (almost like an intro to programming class) and then teach how Lua worked in comparison.
I guess we lucked out because they were able to pick it up quickly after the education on functional programming.
A few of us had some experience with Lua before with game programming, so we took it on ourselves to teach the others Scheme (almost like an intro to programming class) and then teach how Lua worked in comparison.
I guess we lucked out because they were able to pick it up quickly after the education on functional programming.
Torch is really cool. I wish I could use it more. Sadly it isn't supported on Windows.
Initially we had to use Windows for some client requirements, because of that we used Vagrant+Docker to deploy torch. You might want to try that if you can use VMs.
Does anyone know what Facebook is using the AI for?
Basically everything in FB has one main goal: Ads
Secondary goals might be: Spam detection and extracting info from the graph/pictures, etc
FB identify people from your Social Graph in Pictures, for example
Secondary goals might be: Spam detection and extracting info from the graph/pictures, etc
FB identify people from your Social Graph in Pictures, for example
The software runs on Nvidia hardware.
Not sure if all their servers are fitted with one or more GPU cards...
I think it is more likely that this is just research software.
I think it is more likely that this is just research software.
Deep neural networks are usually trained on GPUs which can provide a huge speed up, and is much cheaper. However they can easily be run on CPUs after being trained. I don't see why they wouldn't have access to GPUs in production though.
Well, it depends, you can definitely have servers with NVidia hardware, and you don't need to have all of them with it. (Amazon offers GPGPU servers on EC2 for example)
Amazon only offers very costly Tesla and Quadro cards on their GPGPUs. Super expensive. For scale, if you do not need the extra memory on the Tesla cards, what you want is commodity NVIDIA GeForce cards that are a fraction of the cost.
Standard cards are handicapped for double precision operations though (1/8 speed afaik).
You don't need double precision for typical DL tasks, such as training a convnet.
Double precision is overrated :) (at least for ML)
Also, identifying trends in users engaging in predatory behavior, etc.
"Basically everything in FB has one main goal: Ads"
Is this actually true, or a glib answer? Ads certainly don't feel like much of the FB experience, nowhere near the level of Google, or even Twitter. They're negligible, really.
Whereas shaping the feed and identifying people in pics, etc, is central to the FB experience.
Is this actually true, or a glib answer? Ads certainly don't feel like much of the FB experience, nowhere near the level of Google, or even Twitter. They're negligible, really.
Whereas shaping the feed and identifying people in pics, etc, is central to the FB experience.
> Ads certainly don't feel like much of the FB experience, nowhere near the level of Google, or even Twitter.
Facebook creates an artificial distinction between labeled ads (which are, of course, obvious ads) and posts with paid reach (which are also ads, but presented as normal content), exactly for the purpose of maximizing the quantity of ads while minimizing the impression of content being dominated by ads.
Facebook creates an artificial distinction between labeled ads (which are, of course, obvious ads) and posts with paid reach (which are also ads, but presented as normal content), exactly for the purpose of maximizing the quantity of ads while minimizing the impression of content being dominated by ads.
Right, so ads don't feel like much of the FB experience.
My point is that FB seems much, much more focused on creating an engaging user experience than it does on ads, so it seems wrong to say that their deep learning efforts have ads as their one main goal.
My point is that FB seems much, much more focused on creating an engaging user experience than it does on ads, so it seems wrong to say that their deep learning efforts have ads as their one main goal.
> Right, so ads don't feel like much of the FB experience.
Right, but the issue was the claim that with FB everything is about ads, not the UX feels like it is about ads. Making ads not feel like as much of the FB experience as they are is, itself, about ads -- and, particularly, is about maximizing the degree to which the FB experience is actually about ads.
Right, but the issue was the claim that with FB everything is about ads, not the UX feels like it is about ads. Making ads not feel like as much of the FB experience as they are is, itself, about ads -- and, particularly, is about maximizing the degree to which the FB experience is actually about ads.
"Making ads not feel like as much of the FB experience as they are is, itself, about ads..."
Well, we just disagree here. I see it from exactly the other end. That the main FB goal is to engage users, and minimizing the (necessary, but grudgingly so) ads is about that.
Well, we just disagree here. I see it from exactly the other end. That the main FB goal is to engage users, and minimizing the (necessary, but grudgingly so) ads is about that.
I think the answer is both-and vs. either-or.
FB's bait is user engagement. So, applying machine learning to the goal of engaging users makes sense.
But, to monetize that user-engagement they, of course, use ads. Applying machine learning directly there makes sense too.
Beyond that, it's just semantics. Yes, one could accurately say that everything FB does is about ads because, ultimately, they are a for-profit company which derives its revenue from advertising. In that way, the ultimate goal of all of its activity is aimed at generating ad revenue.
But, that doesn't mean that every single activity or bit of technology that supports that activity is directly tied to driving ad revenue. That is to say, that if FB featured no advertising but, instead, charged users, they would still likely deploy machine-learning to user-engagement and, thus, customer-retention.
FB's bait is user engagement. So, applying machine learning to the goal of engaging users makes sense.
But, to monetize that user-engagement they, of course, use ads. Applying machine learning directly there makes sense too.
Beyond that, it's just semantics. Yes, one could accurately say that everything FB does is about ads because, ultimately, they are a for-profit company which derives its revenue from advertising. In that way, the ultimate goal of all of its activity is aimed at generating ad revenue.
But, that doesn't mean that every single activity or bit of technology that supports that activity is directly tied to driving ad revenue. That is to say, that if FB featured no advertising but, instead, charged users, they would still likely deploy machine-learning to user-engagement and, thus, customer-retention.
What it feels like to the user doesn't matter. They are still maximizing their ad revenue.
What does matter is what their primary goal is- Is it to increase user engagement for its own sake (and ad revenue is an incidental necessity), or is it to increase user engagement to maximize their ad revenue?
That is, is the main goal of Facebook to engage and connect people via its platform, or is it to make money from selling ads?
From everything I've seen from them thus far, I just don't believe ad revenue is their main goal. They do seem incredibly obsessive about engagement, but ads seem like a neglected afterthought.
That is, is the main goal of Facebook to engage and connect people via its platform, or is it to make money from selling ads?
From everything I've seen from them thus far, I just don't believe ad revenue is their main goal. They do seem incredibly obsessive about engagement, but ads seem like a neglected afterthought.
What it feels like to the user is vital. The AI is probably to find the most engaging way to display ads. More engagement with ads == high ad valuation.
Not necessarily Ads in the traditional sense, but FB also sells content reach (Promoted Content) and targeting
Shaping the feed is certainly one part of it. But I guess that part of the success of FB is that they don't make it, as you said, not make the ads feel as much part of the experience.
Shaping the feed is certainly one part of it. But I guess that part of the success of FB is that they don't make it, as you said, not make the ads feel as much part of the experience.
Categorization is critical to boosting precision in information retrieval tasks. Deep networks for cropped object recognition are an important source of (inferred) content categories (especially when other context, e.g. photo captions, are not useful). While a good image categorizer would be very useful for ad targeting and auctions, the technology can also boost precision for news feed, graph search, etc., which improves user experience.
Agreed. You can't show ads unless you have users. Optimizing advertisements is definitely second fiddle to optimizing the user experience.
Someone with more context can answer this better but I would guess that selling behavioral data to ad agencies is a thing?
[deleted]
[deleted]
To further personalise the user feed.
Wow great. I dreamed of an AI platform based on LuaJIT a few years ago. Torch and FB's deep-learning module sound amazing!
For those who aren't aware, it says
1. If facebook sues you, and you counterclaim over patents (whether about software or not), you will lose rights under all these patent grants.
So essentially you can't defend yourself.
This is different than the typical apache style patent grant, which instead would say "if you sue me over patents in apache licensed software x, you lose rights to software x" (IE it's limited to software, and limited to the thing you sued over)
2. It terminates if you challenge the validity of any facebook patent in any way. So no shitty software patent busting!