I don't think OpenAI and Google have no moat. The biggest moat they have is hardware-software integration, which allows them to serve the models at a very cheap price.
1 point by deeplstm 0 minutes ago | prev | next | edit | delete [–]
I don't think OpenAI and Google have no moat. The biggest moat they have is hardware-software integration, which allows them to serve the models at a very cheap price.
I don't think OpenAI and Google have no moat. The biggest moat they have is hardware-software integration, which allows them to serve the models at a very cheap price.
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
https://arxiv.org/abs/1910.13461 (paper)
Modelling long sequences has always been hard for transformer-based models. This paper proposes a super innovative way for the transformer to cache previously processed tokens. And it makes generation 9X faster. This is truly mind-blowing
This is super impressive!! Those generated images are quite accurate and realistic. Here are some of my thoughts and explanation about how they do use discrete vocabulary to describe an image.
https://youtu.be/UfAE-1vdj_E
There are 300+ deep learning papers published every day. I find it very hard to keep up with. This paper introduces a cool way to summarize papers to extremely short summaries (TLDRs).
More interestingly, Semantic Scholar uses the proposed method to build its TLDR feature and it's available in beta for nearly 10 million papers now!
The AI system built by an MIT team can detect it with 97% accuracy. More interestingly, it's able to detect asymptomatic people 100% (sensitivity). The proposed model comprises 4 biomarkers (3 ResNet models and a Poisson mask). Each of them represents a hypothesis of the repository disease.
Caveat: More replication is needed. There are clinical trials ongoing in Mount Sinai and White Planes Hospitals in the US, Catalan Health Institute in Catalonia, Hospitales Civiles de Guadalajara in Mexico, and Ospedale Luigi Sacco in Italy.
How to perform full end-to-end entity linking has always been a challenging problem in NLP. The typical approach for this is to use a model to detect entities and then employ another model to perform entity disambiguation. And this paper beautifully formulates these two steps into a single neural network model.
It's a super cool paper that invents "vokenization" to generate a large amount of visually-grounded language datasets and trains visually-grounded models on those.
Most language models are trained on pure text data. Although it achieves significant success in recent years, this is not how humans acquire a language. It raises an interesting question "Can language models achieve a high level of language understanding by reading the text input alone?" The answer is probably "no".
To push the boundary of language models, adding other learning signals in the learning process is the key to success. And the first thing that comes to my mind is vision (visual cue). However, the existing visually-grounded datasets are a level of magnitude smaller than pure text ones. This paper purposes "vokenization" method to overcome this problem, and uses the new data that generate to train visually-supervised language models.
More importantly, visually-grounded models show significant improvements over text-grounded only models.
This video explains a legendary paper, BERT. It leverages the Transformer encoder and comes up with an innovative way to pre-training language models (masked language modeling). BERT has a significant influence on how people approach NLP problems and inspires a lot of following studies and BERT variants.