Great question! Let me rephrase so you can confirm my understanding: I have some labeling functions (LFs) that are far more accurate on a majority subset of the data than on one or more minority subsets or "slices" of the data... and these subsets are not necessarily correlated with the class labels, so this isn't a traditional class imbalance problem...
We've actually done some recent work on this (https://papers.nips.cc/paper/9137-slice-based-learning-a-pro...) where we have users define these critical "slices" approximately so that the model being trained can pay special attention to them (extra representation layers) so they don't get drowned out by the majority subsets/slices. But definitely a lot more to do in this area!
Agreed! As noted in other answer, Snorkel certainly does not work for everything :) And indeed, in many cases it may be easier to express what you know extensionally (label examples) vs intensionally (write functions). A lot comes down to the unit cost per label over time- and whether it's more economical to label a bunch of data by hand vs. write LFs or similar.
That's btw why a lot of examples of ML today are ones where data is (i) simple for non-experts to label, (ii) non-private and therefore easy to outsource for labeling, and (iii) low rate of change (e.g. images for self-driving, basic NLP stuff for chat bots, etc)- this kind of data can be labeled cheaply and once, so hand-labeled training sets are (barely) economically feasible to build manually. However, most data is not that easy or cheap to label, needs to be relabeled constantly to adapt to change, and thus the investment in a programmatic approach is often far better even if certainly not push-button!
More advanced structured prediction tasks are still definitely on the cutting edge- mainly IMO down to defining the semantics of the programmatic user input like labeling functions for these kinds of tasks. Some recent work (http://cs.brown.edu/people/sbach/files/safranchik-aaai20.pdf) has extended these semantics for sequence tagging, as an example- so some exciting moves in this direction!
Yup! The LFs can also actually be correlated- just not too correlated (roughly, think of needing at least three mostly uncorrelated cliques, to be precise).
Part of the high level description, though, is that a lot of different parts and lines of work are integrated into Snorkel Flow beyond just this original programmatic labeling idea. So also programmatic operators for data augmentation, "slicing" or partitioning of data, and the overall end-to-end platform (UI + SDK) supporting iterative development of ML models via this paradigm of programmatic training data.
Yeah definitely- and would love to chat sometime, as this is a space I've at least had less direct hands-on interaction with. There's a line of work in the ML literature on "Positive unlabeled (PU) learning"--basically, setting where there are only positive labels or abstains--with a lot of theoretical ties to what our stuff rests on, I think a tie in here is interesting. Of course, most of these approaches rely on some (to varying degrees) hidden and very strong distributional assumption... anyway looking forward to a chat!
You can always do multi-label as a multi-task learning model (or just a set of binary models), which is something we (and many others) have explored before! A lot of the adjustments for mainline Snorkel have to do with (A) the semantics of the labeling functions (need to be able to express that something is not class A and/or have a general per-class prior) and (B) all the infra to support what is just now a bunch of independent per-label binary tasks, at base
Lurking for a few more min... great question! Dealing with both class imbalance and issues of pernicious biases in both underlying data distributions and training labels is an extremely important topic. Our underlying theory deals with local biases (e.g. individual labeling functions or sources of training signal being biased) but systemic biases (e.g. the user driving the system being biased) are certainly tougher.
One important and practical answer that we've found: with an approach like in Snorkel Flow, you can inspect the source of the training data and correct it if biased- which you just can't do with e.g. a million hand labeled training data points. So in practice this is a big advantage we've found.
On the theory / research side, this is definitely an area we want to pursue further!
Single label has been the applicable one for most of the applications we've tackled to date, but agreed that multi-label is also very important! More coming here soon...
First: Snorkel Flow absolutely does not generalize to every ML problem :). IMO defining where different systems and approaches do and don't work best is one of the most important and most challenging problems in ML systems research- as noted, we've worked to give detail on this for Snorkel over the years... no perfect answers, but some notes below:
- As you imply, a lot has to do with the available sources of input signal- whether labeling functions, or 'transformation functions' for data augmentation, or other ops we've worked on... the input is obviously key.
- For data modalities like image, video, etc: Often the most successful approach is to (A) rely on some pre-processed features or "primitives" and write labeling functions over these- as my co-founder Paroma in particular has published about over the years- and/or (B) use metadata
- External models are definitely expressable as labeling functions, and we've worked on exactly that problem of modeling (local) biases and correlations!
Thanks for this kind note! Would love to chat sometime and hear about your findings working with Snorkel- pros, cons, feature requests, etc. Multi-label is one on our list and under development (see comments below and elsewhere), would be interested to trade notes!
Hi all, this is Alex from the Snorkel team- thanks for all the great comments! Excited to respond to a few questions directly, but first highlighting some up here:
- Where to find more about the core Snorkel concepts: We've published 36+ peer-reviewed papers, along with blog posts, talks, office hours, etc over the years (see https://www.snorkel.ai/technology and https://www.snorkel.ai/case-studies), so I'll defer somewhat to those... but of course, academic papers can be painful to read (even when you wrote them!), so happy to also answer questions here.
- What Snorkel Flow is: Snorkel Flow is an end-to-end ML development platform based around the core idea that training data is the most important (and often ignored) part of ML systems today, and that you can label, build, and manage it programmatically with the right supporting techniques. This is based on our research at Stanford, where we spent several years exploring the basic question: can we enable subject matter expert users to train ML models with things like rules, heuristics, and other noisy sources of signal, expressed as "labeling functions" and other types of programmatic ops (ex: 'label this document X if it overlaps with dictionary Y'), instead of having to hand-label training data. This type of input, often termed "weak supervision", ends up requiring a lot of work to deal with as it is much noisier than hand-labeled data (eg the labeling functions can be inaccurate, differ in coverage and expertise, have tangled correlations, etc) but can be very powerful if you model it right! And Snorkel Flow specifically is focused on actually making the broader end-to-end process of building and managing ML with programmatic training data usable in production, rather than just on exploring the algorithmic and theoretical ideas as was the goal of our research/OSS code over the years!
- Why train a model if you have a programmatic way to label the data: In Snorkel, the basic idea is to label some portion of the data with labeling functions (usually it's hard to label all of the data- hence the need for ML), and then use ML to generalize beyond the LFs. In this sense Snorkel is an attempt to bridge rules-based approaches (high precision but low recall) and stats learning-based approaches (good at generalizing). This is also useful in "cross-modal" cases where you can write LFs over one feature set not available at inference time, but use them to train a model that does work on the servable/inference time features (e.g. text to image is one recent example https://www.cell.com/patterns/fulltext/S2666-3899(20)30019-2). But, of course, we believe in an empirical process all the way, which is another reason we like the Snorkel approach: if you can write a perfect set of labeling functions, then great- you don't need a fancy ML model, stop there!
- Does Snorkel work??: As an ML systems researcher, I'm always a bit perplexed by this question... the relevant questions for any system or approach are usually 'When/where might it be expected to be useful, and what are the relevant tradeoffs?' We've done our best to answer these questions over the years with theory, empirical studies, etc (see links above), and of course its very case specific. But one thing I'll note is that Snorkel is not a push-button automagic approach that takes in garbage and produces gold. It's our attempt to define a new input / development paradigm for ML--one which we've shown can often be orders of magnitude more efficient--but like any development process, it requires effort and infrastructure to use most successfully! Which is a big part of why we've built Snorkel Flow- to support and accelerate this new kind of ML development process.
- Who uses Snorkel? A few that have a published record: Google, Intel, Microsoft, Grubhub, Chegg, IBM... and many others at very large and smaller orgs that are not public
- What is going to happen with the OSS: The OSS project will remain up and open under Apache 2.0, same as all of the other research work we've put out over the years! See our community spectrum chat for more.
The grouping of theoretical, 'QED' style math with music and football is interesting to think about. I think football and math go together and can be seen as inverse processes wrt to music. Both football and theoretical math have you start with arbitrary rules or principles and then build up a system, add extrapolated layers of complexity on top of that. The top layers recreate the world building up from that core set of assumptions.
Music on the other hand can be seen, at its most elegant, as taking the complexity of the world and reducing it or rather tying it together, coalescing it into patterns, rhythms, and other things that approach the simplicity and fundamental nature of what one starts out with math, or at least the earliest lemmas of a mathematical system.
I think both are intensely pleasurable activities in their own way because they let you trace this path from fundamentals to complexity and back. Math and football as challenges that have you climb up the mountain of complexity, but that still remain close enough to the base to be simple and absolute and beautiful in these ways; and music as an initially complex and tumultuous thing that gets rid of just enough entropy to be simpler but not fake.
To be annoying by tacking a long, overly thought-out comment onto yours (sorry): Agree with this root sentiment, for many other things that used to be done casually and amicably in real life and are now replaced by process-driven, efficiency-focused and sterilized online versions. However I think that such a substitute is ok, or better- great, when (a) the thing just wouldn't happen at all in the first place because it's just too difficult (e.g. say keeping in regular touch with a very large group of far away friends) or (b) it's not pleasant in its real-life instantiation anyways. I think nagging friends to pay you back esp. for small amounts satisfies both of these criteria.
Sounds really cool. I like services that offer solutions that are easier along the dimension of horrible-awkward-interactions. I.e. I think that maybe the biggest appeal of this is not actual the time saved finding the person at a time when they have the right cash on hand, but the fact that Mint will send the email for you in a clean, impersonal but not menacing manner.
I haven't tried this or Venmo but I think it would be cool if they added some sort of group portal service, where you could email a group of friends, say for "Dinner: $96", and then Mint would email all of them with a link to a spreadsheet/form of some kind where they could list what they've already paid each other, other related grievances that would affect how much they owe, etc. And then once all had approved the resolved list of payments, the process would proceed on an individual level as normal with the correct amounts.
Haha. Someone should do a web scrape of all parkour club directories, and then look at the intersection of that set with Boston Dynamics' team... could explain some things
I should caveat subsequent comments by saying that I would have given my childhood for this toy (and therein I would have learnt the meaning of irony too, but I digress).
Anyway I am wondering what this could really be intended for. If it is indeed meant to carry a camera payload for cheap soldier surveillance, then I have an alternative suggestion: http://www.amazon.com/Parrot-AR-Drone-Quadricopter-Controlle.... This $298 quad-copter with two mounted cameras seems like it could do the job just fine. (Incidentally on the subject of cheap RC toys saving soldiers' lives and proving themselves viable short-term options: http://www.rccaraction.com/blog/2011/08/04/traxxas-rc-truck-...). Anyway, especially with Petman (http://www.bostondynamics.com/robot_petman.html) purportedly being for chemical suit testing (which in my opinion is like saying crash dummies need to have guns and machine vision), Boston Dynamics has shown that it likes releasing potentially mid-path projects for other-than-ultimately-intended purposes (makes perfect sense from a testing perspective). So I wonder what Sand Flea could be working towards other than surveillance? 11 lbs./30 ft is not bad considering a quad copter of similar size would be probably max 4-6 lbs. So maybe they are gearing it up for heavier payloads. Nothing like the 400 lbs alpha dog carries (at that point, assuming similar tech, the Sand Flea would have to be big enough to basically just roll over single-story buildings), but perhaps an explosive payload or something in the neighborhood of 10-20 lbs.
The patents are assigned to NASA (see for example: http://assignments.uspto.gov/assignments/q?db=pat&pat=75...). If NASA had been funding an outside contractor they would have had a non-exclusive non-transferable 'walk-in' license to the patents; however they must have invented them in-house and so they own them.
One could argue that even though NASA = the gov't funded the R&D that resulted in these patents, they don't need the monopoly rights of patents because if others then take their ideas, no loss!- it could just be considered stimulus spending, which is arguably part of the government's mandate anyway and thus not just a net loss of money on the budget balance sheet...
However, and in my opinion far more importantly, there is a nice incentive structure at work here to the general benefit, which is that by allowing NASA to own and sell intellectual property- but only via the public patent system- you give them a monetary incentive to disclose their patents and put the technology behind them into the public domain. If they had no rights to own the technology, then they would have no clear incentive to ever release it (which is what government agencies do anyway with many top secret technologies- these it does not own exclusivity to, which might be what you were thinking of). However this way, there was an incentive for NASA to file a patent, it did, and people could have been doing incremental/additive R&D based on the publicly disclosed invention spec for the past three years. "On the shoulders of giants" is still a decent credo generally, even (or especially, depending on your perspective) if you have to pay said giant a small licensing fee sometimes...
(Of course there are other huge problems with the patent system but in my opinion this fundamental tenet of creating an incentive structure which encourages (a) invention and (b) disclosure is still- theoretically at least- a vastly good thing for us)
Cool article! I think a broader question raised is one I have thought about a lot: what is the optimum trade-off between narrow clarity of function/purpose and broad ambition/expansiveness/spread of features? On the one hand: we obviously admire and use things that are ambitious and full-featured (google, facebook, etc. etc. etc.) On the other hand though: HS points out that the narrowed "X for Y" model makes a better YC pitch... but more importantly, it might make a better consumer pitch. E.g. sometimes I wonder why I don't use wolfram alpha more often... and I think one of the answers is that I don't quite know what it does. It possibly covers TOO much area and I might- at a moment of quick snap judgement- pick the far more limited service with the easily comprehensible/memorable single feature claim... Thoughts?
What I don't quite understand about this piece is the motivation for saying that the biggest value-add would be to 'kindly inform the person that the correct term is 'co-founder'" (I am purposely trying to keep this comment symmetrical by leaving out "technical" here...). Why would it be valuable or even 'charitable' to do this? Them calling programmers 'programmers' or vice-versa has no negative effect for the named person- IN FACT it has a large positive one as it marks those people who don't have proper respect for their counterpart co-founder role. 'Kindly informing' the transgressor of such a misphrasing is like giving them the secret password without them having earned it themselves via a proper, self-driven cultivation of proper respect for their counterpart role.
We've actually done some recent work on this (https://papers.nips.cc/paper/9137-slice-based-learning-a-pro...) where we have users define these critical "slices" approximately so that the model being trained can pay special attention to them (extra representation layers) so they don't get drowned out by the majority subsets/slices. But definitely a lot more to do in this area!