I basically share your sentiment. However, Greg Yangs work seems to have produced something of direct practical benefit for training large neural nets, based on the NTK literature. µ-Parametrization is apparently very useful in real world practice.
Yes, it was. From their site: "It has been trained on a selection of English language and source code from publicly available sources, including code in public repositories on GitHub."
I think they would in any case. My impression is that data is siloed internally at Google, and that data sharing between departments would be way more complex than just setting up some (possibly redundant) logging.
Similar to how "god of the gaps" arguments work, there's a "god of complexity" phenomenon in tech, where people will project infinite possibilities onto technology they don't understand.
Blockchain is a brilliant solution to an extremely rare problem. It lets you do distributed consensus among untrusted nodes in a setting where contributing compute power is economically incentivized (e.g. where there's mining). That applies to cryptocurrencies, but basically nothing else.
It's amazing how much bullshit has been pitched under the flag of "blockchain". Millions of dollars have been poured into projects that are technically unsound or that had no use for a blockchain to begin with. This has been going on for years.
You have people born in the 90s hospitalized in Norway now. Many experience long term physical and cognitive effects. And as long as we're assuming everyone goes unvaccinated, the chance of contracting COVID goes towards 1 over time, while the R-number explodes as we open up society to avoid economic ruin. Not a good scenario IMO.
As I said, the relevant comparison in the real world is versus waiting for another vaccine, where it the choice is much less obvious, and dependent on the disease level in the population.
I'm happy this research has been done so quickly, but am somewhat annoyed that the public conversation is so focused on whether or not the vaccine can possibly cause any problems, and not at all focused on the risk management question of what the optimal choice here is.
The chance of getting a blood clot seems so low that it may for all I know be equivalent to the risk of dying in the streets while jaywalking to get to the vaccination center. Compared to the risk of going unvaccinated, it's an obvious choice.
That's not the choice we have though. The correct comparison seems to be the expected risk of taking the vaccine vs expected risk of waiting for a dose of another vaccine. Calculating this on an individual level should be straight forward given the risk of contracting COVID in a given area. Ideally the calculation would also take into account the effect that vaccination has on the virus reproduction number. That's harder to model, but approximate models like this exist.
If you actually put numbers to it and do this computation, you'll get an answer, or at least a distribution of outcomes, where it's straight forward to see what choice is optimal. Then public discussion can be around what the parameters of this model should be, and we'll stand a chance of making the right choice.
This is a potentially very consequential optimization problem, and the public discussion about it is as if no one understands that that is what it is.
Anyone can go on archive.org and verify that they changed their color scheme and header to ones very similar to CatchJS in Feb 2020, after CatchJS had used that look for 2 years. Thank you for also pointing out the pricing page, where the same is true.
I've now received a cease and desist from their attorney stating that the looks are too similar. It's nice that they've paid for an attorney to make my point for me, but they seem to have mixed up who did what here.
A middle ground could be nice: An IDE extension that notifies you when something you're writing will conflict in the future, should you and your coworker both commit and push what you've typed out. It would allow you to sort that out immediately, or at least plan ahead, rather than being surprised by a large merge conflict n days down the road.
If you talk to Norwegians about Norwegians, you'll discover that we're very good at patting ourselves on the back. You might say that we're one of the best countries in the world when it comes to patting ourselves on the back.
The problem isn't having an MBA though is it, it's (allegedly) not being competent enough in engineering. All the CEOs you mentioned have engineering degrees, while Intel's CEO does not.
That's an excellent example to show why statistical NLP outperforms discrete parsing using logical rules. That A1 and A2 means the same is something that is clear to us from the context. This is something a continuous vector space model can capture and a discrete rules based model cannot.
The difficulty of the problem is that of a 50-way classification. If the only goal was to minimize WER, a simple post-processing step choosing the nearest sentence in the training set could easily bring the WER down further. They've chosen to do it the way they did it presumably to show that it can be done that way, and I don't fault them for it.
They claim that word-by-word decoding implies that the network has learned to identify words. This may well be true, but it isn't possible to claim that from their result. For example, let's say you average all electrode samples over the relevant timespan, transform that representation with a FFW neural net, and feed that into the an RNN decoder. It would still predict word-by-word, on a representation that necessarily does not distinguish between words (because the time dimension has been averaged over). Such a model can still output words in the right order, just from the statistics of the training sentences being baked into the decoder RNN.
This is cool. For those who are not super familiar with language processing, I think it's good to point out the limitations of what's been done here though. They mention that professional speech transcription has word error rate around 5%, and that their method gets a WER of 3%. Sure, but the big distinction is that speech transcription must operate on an infinite number of sentences, even sentences that have never been said before. This method only has to distinguish between 30-50 sentences, and the same sentences must exist at least twice in the training set and once in the test set. Decoding word-by-word is really a roundabout way of doing a 50-way classification here.
It's an invasive technique, so they need electrodes on a human cortex. This means data collection is costly, so their operating in very low data regime compared to most other seq2seq applications. It seems theoretically possible that this could operate on Google translate level accuracy if the sentence dataset was terrabyte sized rather than kilobyte sized. That dataset size seems very unlikely to be collected any time soon, so we'll need massive leaps in data efficiency in machine learning for something like this to reach that level. They explore transfer learning for this, which is nice to see. Subject-independent modelling is almost certainly a requirement to achieve significant leaps in accuracy for methods like this.
> They only had 36 samples, and a 3sd event occurs once in ~300 cases: this hints to the fact that that data point might have been erroneous
Or it hints that the distribution of learning rates is not gaussian. When there's an "n-sigma" event, it's usually much more likely that the model is wrong than that the event is that rare.