Access to parallel corpora is a limiting factor in general. A good way to train a language translator is to use an open source dataset (several here http://opus.nlpl.eu/) to train a base model, and then fine-tune it with a smaller dataset specific to your domain.
In this case, the author claims pretty good accuracy, almost on par with Google Brain's!
On my test set of 3,000 sentences, the translator obtained a BLEU score of 0.39. This score is the benchmark scoring system used in machine translation, and the current best I could find in English to French is around 0.42 (set by some smart folks as Google Brain). So, not bad.
A GAN style approach to learning and generating variants could be interesting as well. It could generate a couple of hundred plausible versions. Then you have another network that is trained to differentiate between fake and real colored photos which picks the best version.
Agreed. I imagine this has applications in compression as well. You could stream a movie (or a football game) in black and white and enable each device to color it on the spot. A similar technique could also be done for HD/3D/VR.
Hey, thanks! I'm one of the co-founders at FloydHub.
Our focus has been on individual data scientists thus far. We'll be rolling out a teams plan soon since folks doing deep learning at work (with on-prem or cloud infra) tend to run into more or less the same difficulties as individuals. If you have any feedback, I'd love to hear :) sai (at) floydhub.com
FloydHub | Cloud Infrastructure Engineer | San Francisco, CA | Onsite | Full-time | Salary: 100k-125k (0.5%-1% equity) | https://floydhub.com
FloydHub (YC W17) is building a Heroku for deep learning. We enable data scientists to do deep learning in the cloud with a few simple commands and without any of the infrastructure or DevOps hassles.
Every day, we handle training, scaling and serving of several thousand deep learning models on our GPU clusters and manage TBs of data. As an infrastructure engineer, you will be responsible for building and scaling our GPU cloud infrastructure. Not simply operating the system, but be part of architecting and building scalable and secure cloud infra.
Requisites: 3+ years of cloud infrastructure experience
We're a small team (4 engineers), agile and very early stage (YC W17). We're Stanford/CMU grads, with experience leading deep learning research at Bing/Microsoft, large engineering org (Location Labs, Avast) and infrastructure at LinkedIn. We're backed by some of the best VCs and angels in town (https://floydhub.com/about). If joining an awesome 4-person team doing AI/infra excites you, come join us! As a founding team member, you will have the opportunity to considerably impact not only the technical direction of the product, but also the culture of the company!
DevOps is indeed a huge bottleneck in deep learning. Provisioning machines, installing drivers and packages and managing their dependency hell distracts focus from the core deep learning. At FloydHub (I'm a co-founder), we're building a zero-setup deep learning platform.
Spinning up a Jupyter notebook with Pytorch 0.2 is as simple as `floyd run --env pytorch-0.2 --mode jupyter`. All the steps you mention in your comment are automated.
DevOps hassles is, of course, just the first of many hurdles to doing effective deep learning. Experiment management, version control, reproducibility, sharing & collaboration, etc. are also other important problems.
FloydHub (YC W17) is building a Heroku for deep learning. Data scientists can train and deploy deep learning models in the cloud with a few simple commands and without any of the DevOps hassles. Instead of worrying about provisioning GPUs, installing drivers, and managing software dependencies, focus on what matters - the science itself.
We're small, agile and very early stage (YC W17). We're backed by some of the best VCs and angels in town. If joining a 4-person deep learning/infra startup excites you, come join our core founding team!
- Help us grow 5x, 10x and 50x our current scale. You will lay foundations for our community building initiatives. Expertise: data driven growth mentality
Hey! Glad you brought this up. We've gotten quite a few requests from students enrolled in Udacity courses and have been helping get them set up & run their class projects on Floyd.
You mean bash on Windows? The CPU version of dl-docker will work on Windows! But unfortunately, the GPU version does not. You would need GPU passthrough, which is currently not supported. See https://github.com/NVIDIA/nvidia-docker/issues/197
We’ve definitely felt the bane of forgetting to turn off some really expensive GPU instances. Efficient scheduling and spinning instances up/down as required is one of the first things we built to cut down on the costs.
Git is an apt analogy. The search space of hyperparameters is usually fairly large for most DL algorithms, so a good amount of experimentation is required to tune them. Things can start to get haywire without end-to-end version control of code, data, parameters, results, environments, etc. Definitely one of the core problems we solve.
That’s awesome! ML to some extent, and DL to a large extent are still not considered mainstream enough by many. The big players (Google, Facebook, Baidu) seem to recognize this and invest appropriately, but the others not so much yet. That’s hopefully a sign of a burgeoning market!
Defending against big players with almost infinite resources is always an interesting problem. Kind of have to hope they don’t come at you head on :) That said, there might be other aspects that come into play. For example, the market is fragmented with big providers each having their own frameworks and infra that they prioritize (Google - Tensorflow/GCP, Facebook - Torch, Microsoft - CNTK/Azure, Amazon - MXNet/AWS, Baidu - PaddlePaddle, etc.) Since there is yet no clear winner, which will likely be the case for the near future, vendor lock-in is not really desirable. We intend to be backend / framework / language agnostic.
I’ve heard FBLearner Flow is pretty cool for running/managing/sharing ML pipelines inside Facebook. Never seen or used it myself, but Microsoft had a similar internal tool called AEther that was very cool too. We’ve definitely taken inspiration from AEther in building Floyd.
Here’s an anecdotal story about how awesome AEther was (been a long time, so a little fuzzy on details): In 2011, Harry Shum was the VP of the Bing division at Microsoft. It was the early days of Bing (~10% market share, ~$2bn annual loss, etc.) - we had good talent, but were lagging behind Google in tech. In one of our all-hands meetings, Harry jokingly announced that if we beat Google in our core relevance metric (called NDCG), he’d take the entire Bing team, approx. 300 people strong, for a fully paid trip to Las Vegas.
Sure enough, a year later, Bing did beat Google in our core relevance metric (http://www.insideris.com/microsoft-bing-beats-google-in-the-...) and all 300 of us went to Vegas for a weekend as promised. (Spoiler: Google did eventually beat Bing back later)
The success and rapid acceleration in relevance gains was attributed in large parts to the introduction of a new tool called AEther (in addition to improving ML tech and hiring top talent). AEther was an experimentation platform for building and running data workflows. It allowed data scientists to build complex workflows and experiment in a massively parallel fashion, while abstracting away all the engineering concerns. I used it a ton on a daily basis and loved it. The AEther team claimed that it increased the experimentation productivity of researchers and engineers by almost 100X. Even now, when I ask ex-Bing data scientists working at other companies about what they miss the most from their time at Microsoft, AEther is almost always in the top 3 answers.
Having seen how awesome AEther was from the inside, one of our goals is to bring its benefits to the rest of the world as well. However, having talked to a few individual data scientists and researchers over the last month, their preference seems to be CLI over GUI (while bigger companies like it much better). May be its one of those things you have to get used to, or may be our implementation is clunky. So we’re making the GUI an enterprise only feature for now, while we continue to help individual data scientists through our CLI.
Haha, there’s no magic and it’s difficult to say with any certainty that we’re going to make it. When we started out, we were just scratching our own itch. The AI community is amazingly open and fast paced. May be because of that, the tooling around it isn’t as mature as for, say, software development. More companies, IMHO, tend to be focused on building the next big thing _with_ AI, rather than the next big thing _for_ AI. With the increasing popularity of the space and the relative infancy of the state of tooling, we believe now might be a good time to tackle this problem.
That said, it’s hard to say how things are going to turn out. This is my first startup after working in the corporate for 6 years and it’s been amazing so far. Learning a lot, and excited for what’s ahead! :)
Thanks for the comment! There's lots of challenges to be solved in this space, and I'm sure there's room for all of us. Excited to see what you guys are up to. I will look forward to your beta release "real soon" :)
In this case, the author claims pretty good accuracy, almost on par with Google Brain's!