- There is no "moat" (lasting, easy-to-defend technological edge) in AI model businesses. There are just short-term advantages.
- An AI business is a capital-intensive business, just like old factories. Data centers are expensive, models are energy-hungry, and the hardware inside must be replaced every 3–4 years.
- Smaller, specialized models eat margins from below. Transcription, voice, or image detection do not need large models.
There is no reason to expect high margins like you can in traditional software business. Benefits of AI go mostly to consumers.
edit: There is potential for economies of scale. Few megacorps can strive for cost advantage when they achieve scale (Microsoft, Google, Amazon and Meta)
If you take a birds eye view, fundamental breakthroughs don't happen that often. "Attention Is All You Need" paper also came out in 2017. It has now been 7 years without breakthrough at the same level as transformers. Breakthrough ideas can take decades before they are ready. There are many false starts and dead ends.
Money and popularity are orthogonal to pathfinding that leads to breakthroughs.
Without looking the answer, what is your intuition about the size of the VC-dimension of ReLU networks as a function of a number of weights and layers?
That's is based on old assumption of neuron function.
Firstly, Kurzweil underestimates the number connections by order of magnitude.
Secondly, dentritic computation changes things. Individual dentrites and the dendritic tree as a whole can do multiple individual computations. logical operations low-pass filtering, coincidence detection, ... One neuronal activation is potentially thousands of operations per neuron.
Single human neuron can be equivalent of thousands of ANN's.
> deep learning architectures have been crafted to create inductive biases
matching invariances and spatial dependencies of the data. Finding corresponding invariances is hard
in tabular data, made of heterogeneous features, small sample sizes, extreme values
Transformers with positional encoding have embeddings are invariant to the input order. CNN's have translation invariance and can have little rotational invariance.
It's harder to find similar invariances to tabular data. Maybe applying methods from GNN's would help?
Effect of exercise for depression: systematic review and network meta-analysis of randomised controlled trials.
Conclusions Exercise is an effective treatment for depression, with walking or jogging, yoga, and strength training more effective than other exercises, particularly when intense. Yoga and strength training were well tolerated compared with other treatments. Exercise appeared equally effective for people with and without comorbidities and with different baseline levels of depression. To mitigate expectancy effects, future studies could aim to blind participants and staff. These forms of exercise could be considered alongside psychotherapy and antidepressants as core treatments for depression.
COWS FLY LIKE CLOUDS BUT THEY ARE NEVER COMPLETELY SUCCESSFUL.
These are from MegaHal that entered 1998 Loebner Prize Contest. MegaHal was able to produce mind-blowing insightful sayings but most were just bs.
It seems that creativity is easy for computers. Just push randomness through some generative algorithm. Curating and selecting the best output makes all the difference. The ability to select, critique, and understand what is generated and what the meaning is is much harder.
Being a universal function approximator means that a multi-layer NN can approximate any bounded continuous function to an arbitrary degree of accuracy. But it says nothing about learnability and the structure required may be unrealistically large.
The learning algorithm used: Backpropagation with Stochastic Gradient Descent is not the universal learner. It's not guaranteed to find the global minimum.
Transformers solve compositional reasoning tasks by reducing multi-step compositional reasoning into linearized subgraph matching without problem-solving skills. They can solve problems when they have reasoning graphs in the memory.
* heard ~220 million words, talked 50 million words.
* read ~10 million words.
* experienced 420 million seconds of wakeful interaction with the environment (can be used to estimate the limit to conscious decisions, or number of distinct 'epochs' we experience)
From a machine learning perspective human life is surprisingly small set of inputs and actions, just a blip of existence.
The next 'move' for cheaters is to use chess computers in a way that passes 'Chess Turing Test' and makes cheating indistinguishable from normal human play under analysis.
When there is money in the game, there is incentive to cheat.
> The report says dozens of grandmasters have been caught cheating on the website, including four of the top-100 players in the world who confessed.
There are probably smart cheaters already playing who are able to evade detection.
Old AI is today's bleeding edge computer engineering. There is an enourmous amount of free lunches for computer engineers and software startups in the old school artificial intelligence.
* modern SAT solver performance is impressive. They can solve huge problems.
* Writing a complex systems configurator with Prolog or Datalog can be like magic.
* Expert systems. There has never been so much use for them than today. Whenever you see expensive systems utilizing complex mess of "business logic" and expensive consultants, you should know there is a better way.
(I use SAT-solvers to partially initialize neural network parameters).
>... Existing methods are computationally expensive as they rely on heavy candidate sampling coupled with scoring, ranking, and fine-tuning steps. We challenge this paradigm with EquiBind, an SE(3)-equivariant geometric deep learning model performing direct-shot prediction of both i) the receptor binding location (blind docking) and ii) the ligand's bound pose and orientation. ...
- There is no "moat" (lasting, easy-to-defend technological edge) in AI model businesses. There are just short-term advantages.
- An AI business is a capital-intensive business, just like old factories. Data centers are expensive, models are energy-hungry, and the hardware inside must be replaced every 3–4 years.
- Smaller, specialized models eat margins from below. Transcription, voice, or image detection do not need large models.
There is no reason to expect high margins like you can in traditional software business. Benefits of AI go mostly to consumers.
edit: There is potential for economies of scale. Few megacorps can strive for cost advantage when they achieve scale (Microsoft, Google, Amazon and Meta)