What I find most exciting here is that they were able to improve performance by combining the three previous steps into one, in other words asking directly for what they wanted rather than trying to trick Hive into executing it in a certain manner.
Presumably the task had been split up in the past to work around performance limitations of Hive; the fact that they got better performance in a single step shows that the tooling is improving and much less complexity is required to implement this kind of job.
That is a good origin story! I would contend though that it was the performing of the data analytics on himself (which helped keep it top of mind and helped him to develop an internal model) that was the most helpful. People simply using the app may find it less so. (Not trying to dump on the product, merely giving my thought processes when I decided not to in the past).
I'm a diabetic (type one, since 1988) who has also been doing ML startups for the last 15 years. My HbA1C scores have always been below 6, controlled with a two to four blood test per day and long/short (currently Novorapid and Lantus) insulin regime. I've always been sporty, and have done some more difficult things like hiking to Everest base camp without a porter, cycling from Montreal to Lake Placid and back, and a few 24 hour rogaining events. In other words, broadly I've managed my diabetes quite well.
I've always been tempted to do something with ML and the datasets I collect off my own body, but I have never seen this point. In my experience, the thing that allows for good diabetic control is a doctor who can help a person understand their own body's reaction as a dynamic system, rather than a rigid schedule or an external system that tells them what to do. A product like this would be most useful in my opinion as a learning aid.
The place I see this kind of data being extremely useful is in the insurance industry. I currently pay a 400% premium on my life insurance due to being diabetic, but my control is extremely good. Were it to allow me to obtain reasonable insurance coverage, I'd be more likely to be willing to go through the data collection headaches involved with this kind of product.
Most teams I have seen have either template scripts or boilerplate that generates datasets, and share both the generated data and the scripts via normal ways that people share data and code: disk, S3, github, emailing of notebooks, etc.
It requires a fair amount of set-up, but works surprisingly well once there is a core team and problems established.
We are building mldb.ai to help bring the data and the algorithms for ML together in a less ad-hoc manner and to help move things out of research and into prod once they are ready. Many of the hosted ML solutions (Azure ML, Amazon ML, Google Data Lab, etc) and other toolkits (eg Graphlab) are working on similar ML workflow and organizational structure problems.
The Tensorflow code mentions "GCUDACC" in several places, and from the surrounding comments it seems to be targeted at OpenCL as well as CUDA. So it seems that this has been at least considered.
It's very hard to find a viable business model in machine learning or AI tools and platforms at the moment: the size of the market is small, and most money is being made by the end users. As a result companies who could get value from AI want to own it themselves (to keep all of this money themselves). It's a little bit like everyone wanting to own their own data center full of servers in 1999; at some point it will seem silly, but it's hard to sell against because it's accepted wisdom.
Doing services is a reasonable response to that market dynamic, although they certainly had a product vision too.
Without knowing any of the details, I'd say they can be proud of what was accomplished and how they managed their exit:
1. From what I've heard, some of the work they did in the medical domain really did help save lives or make them better
2. They probably returned the capital to their investors and kept most of their employees
3. They didn't get bought by a company that would be horrible to work for or is in terminal decline
Credit where it's due: they entered a rough market and managed it well.
In my experience, the architecture supports cards with shader model >= 3.0. Occasionally a commit will break the support (eg https://bitbucket.org/eigen/eigen/commits/a19653b8035d8ace53... was required earlier this year) but this is a function of the speed of development and usually straightforward to fix.
Better, if the processes are on the same machine you could use it to share the data via shared memory or a common memory mapping, to avoid having copies of the data on each end of the pipe.
Once the open source version of Tensorflow releases multi-node support, this would be one way to make it work. There are potential gains from using a GPU for RF training. As for distributing, in my experience for small models it doesn't make much difference and for larger models the cost of distributing the dataset dominates the benefit from having multiple nodes. But an implementation carefully designed for a given node topology could be made more performant.
There is a whole other world of non stochastic gradient descent based algorithms out there; IMO Tensorflow is sensible to stick to one class of algorithms and do it well.
(Disclaimer: I work on mldb, one of the tools on that list).
For profiling of models, almost everything needed is already there. You only need to pass in a StepStatsCollector through the Session::Run() method (I called it RunWithStats() ) and hook it up to the Executor Args by filling in this variable: https://github.com/tensorflow/tensorflow/blob/master/tensorf... You then get a very usable set of profiling statistics out by aggregating the StepStats object. For profiling individual ops on the CPU, perf is very useful.
It seems that MLDB would be a decent fit for this use-case. You would be able to do pre-processing in the background continuously, and predictions could do a significant amount of work on-demand. Depending upon the size of the overall training set, you might need to spin up a larger server for an hour or so to retrain a model... but if you set the system up right, the model would only need to be trained infrequently as most of the work would be done online. The more preprocessing you can do in the background, the richer and smaller the data that would go into the training phase.
MLDB can memory-map some kinds of datasets which would also help with the low memory-to-datasest size ratio.
Please feel free to reach out (jeremy at datacratic) if you'd like to discuss further.
Could you describe your use-case? This is an interesting question and I'd love to hear more about what you are thinking of.
There are two main parts to most machine learning workloads: training and prediction (though in online scenarios they can be mixed). The training is what tends to be memory intensive, as it involves looking at historical data. Prediction, especially in real-time, is typically less memory intensive and more suited to your scenario. It is possible with MLDB to have training and prediction on separate instances to optimize hardware cost.
Our focus to date has been on getting large real-world use cases to fit, for both training and prediction, on a single commodity server with decent amounts of RAM (4GB and up), especially for training. Performance there is good; you can train user scorers or recommenders over hundreds of millions of users with 20 billion historical actions in minutes on a single server. We haven't focused much on the absolute memory footprint and there is some overhead; you will need a server with a good amount of memory, a couple of gigs, to have a pleasant user experience.
(edit): The team tells me it will work decently down to 256MB of RAM or so, for carefully designed automated workloads. For interactive use you will want more RAM.
I would still expect it to tend towards the normal distribution across a large set of documents. If you model positive and negative word counts as a binomial distribution, you have the the difference of two samples from different binomial distributions which would still tend towards normal (I think, though I'm not 100% sure, certainly it's true within my experience). A logarithm would skew away from positive to negative sentiment and is undefined for negative values.
You are right; this does just shift the bias, which is sometimes all you need (you have a simple algorithm, presumably for a reason).
I did misunderstand that you don't have a training set, just a list of positive and negative words. You could still apply a similar idea.
You could test your hypothesis that the score is biased by looking at the average number of positive and negative words per document, and slightly modify your factors. For example if you found that the average document had 6 negative words and 4 positive words, but you think that the average sentiment is neutral across your documents, you could multiply the positive word count by 1.5. It's a less brutal way to accomplish a similar outcome without increasing the complexity of your algorithm.
Otherwise, you will need to use an algorithm that has more discrimination power, and this will likely mean you need a training set. You can go very deep down that rabbit hole, but I would consider starting with Naive Bayes which is essentially learning a weight per positive and negative word and combines them in a similar manner to how you're doing so now. It has the advantage of being a simple algorithm.
1. Insert each negative example six times into your training set (or weight negative examples accordingly, ie use #positive matches - 6 * #negative matches / (2 * positive word count) as your score
2. Take your distribution of sentiment scores as calculated over held out data (or the training set itself, but be warned that this will skew your results), and calculate the mean and standard deviation. Normalize your results by subtracting the mean and dividing by the standard deviation. You can then say that positive sentiment is > 0 and negative sentiment < 0, with the absolute value being the strength of the classification.
Presumably the task had been split up in the past to work around performance limitations of Hive; the fact that they got better performance in a single step shows that the tooling is improving and much less complexity is required to implement this kind of job.