My two cents on paid search engines: yes so much awareness around privacy these days. so there might be a potential shift in user behavior, which opens up a big market. however, i am skeptical because people are used to free search engine that surfaces good results.
so privacy is great value prop. but not so much that i am willing to pay for it while living with worse search quality.
is the "retrofit" strategy living in the past? living in 2021 it seems a bad choice to buy gasoline cars. Most new cars coming out will have some kind of driving assist (L2 autopilot).
I wonder how clubhouse would monetize its traffic. most social networks had to make the advertisement presented in similar ways to the content, e.g. twitter, instagram, tiktok. but I can't imagine clicking on and listen to a "conversation" that advertises a product...
> Regardless, leaders and technology evangelists in this field still preach that by 2025, the overall percentage of the pie for voice will end up being about 15%.
> “GPT-3″ is just a bigger GPT-2. In other words, it’s a straightforward generalization of the “just make the transformers bigger” approach
Yes it’s true. But there is a difference between what’s interesting and what works. deep learning (RNNs, transformers, etc.) is usually old ideas applied at large scale with slight modifications. Proving a model works well at large scale (175B parameters) is a great contribution and measures our progress towards AI.
> The following year, the lab hired Ritchie, a Harvard grad whose father was a Bell Labs alum, and Ken Thompson, a self-described army brat who, by his own admission, spent at least a month actively dodging the Bell Labs recruiter on the Cal-Berkeley campus
> In our experiments with Transformers, we observed that not all the attention heads utilize their attention span to the fullest. In fact, in a task of character-level language modeling, most of the heads were using only a small portion of their attention span. If we can take advantage of this property during training, we can reduce the computation time and memory footprint significantly
tl;dr most of the time science is incremental innovation. and big discovery comes as paradigm shifts, usually first started by a group of “non-mainstream” researchers at the time.
> Hinton has said that when he was growing up, his mother gave him two choices: “Be an academic or be a failure.” His family tree is branch-breakingly weighted with scientists. His great-great-grandfather was George Boole, founder of Boolean logic, familiar to anyone who has done a “Boolean search.” One of George Boole’s sons-in-law was Charles Howard Hinton, Geoffrey’s great-grandfather, a mathematician and sci-fi writer who coined the concept of a “tesseract” (a four-dimensional object we can see in the 3-D world as a cube—well known to all readers of the classic children’s novel A Wrinkle in Time), and who ended up in the U.S.
Interesting fact from reading this that normally gets neglected in media.