Neural architecture search (NAS) is a thing! But it's almost exclusively based on meta-gradients. Again, wouldn't put my money on GAs ever outperforming gradient-based methods again.
An AlexNet/ResNet-type moment may be in the cards for GAs, but I wouldn't put any money on it. They're typically only one better than brute-force. This can be good enough (and is certainly easy to implement), but if you can get a gradient for your problem -- you should use that. And nowadays, you can typically get a gradient!
Nature can't SGD through genomes but has a metric ton of time, so evolution might be near-ideal for sexual reproduction. We typically don't have billions of generations, trillions of instantiations, and complex environments to play with when optimizing functions... It's telling that the fastest-evolving biological system (our brain!) certainly doesn't employ large-scale GA; if anything, it probably approximates gradients via funky distributed rules.
EDIT: The most modern application I can think of was some stuff from OpenAI (https://openai.com/blog/evolution-strategies/). But the point here is one of computational feasibility -- if they could backprop through the same workload, they would.
For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages.
Finally, buying a local cluster of TITAN Xs is an outright weird recommendation for massive models. VRAM limitations alone make this a losing proposition.
Hard disagree. V100s are a perfectly valid comparison point. They're usually what's available at scale (on AWS, in private clusters, etc.) because nobody's rolled out enough A100s at this point. If you look at any paper from OpenAI et al. (basically: not Google), you'll see performance numbers for large V100 clusters.
> gradient descent no longer has to be written by hand
Nobody's been writing derivatives by hand for 5+ years. All major frameworks (PyTorch, Tensorflow, MXNet, autodiff, Chainer, Theano, etc.) have decent to great automatic differentiation.
The differences and improvements are more subtle (easy parallelization/vectorization, higher-order gradients, good XLA support).
Mainly because it is genuinely exhausting for any medical practitioner. That lots of patients "enjoy" googling symptoms and coming up with far-fetched self-diagnoses is a given. But couple that with the perceived intellectual superiority of (software) engineers and you get a recipe for disaster. It's the equivalent of a doctor leaning over your shoulder while you're coding and telling you to remove random keywords.
For this particular problem, Mask R-CNN would have been the way to go -- it spits out instances as opposed to just deciding, for each pixel, to which class it belongs. Or an SSD (if we don't care about the mask at all).
There's many perspectives on everything. Deep ConvNets, for instance, can be expressed as a continuously evolving ODE. Here's a fantastic paper on this view:
Their applications are tabular data (for which MLPs have never been the method of choice) and MNIST (which I could classify at 85% with a rusty nail), so it's not super impressive.
NNs and the associated toolkit shine with structured high-dimensional data where CNNs, RNNs, or modern shenanigans like Transformer networks excel. I sincerely doubt that these networks turn out to be reducible to polynomial regression in any practically useful sense of the notion. But who knows.
This represents the absolute worst kind of science journalism, completely devoid of context and domain knowledge. Virtually every definitive statement in here is wrong. Their explanation of spiking alone is a complete disaster.
Out of all the modalities, vision is easily the one we know the most about. And we do so at a fairly deep level. The discussed work seems fine but it's not the groundbreaking insight it's made out to be. Great PR work from the involved scientists (or their enterprising university marketing department).
Running Windows is perfectly fine; the major libraries for GPU-accelerated autodiff and networks (CUDNN with Pytorch or Tensorflow) have great support nowadays. It's the AMD GPU that remains essentially useless, as of 2019. If you want to get into the game, I'd recommend buying a middle-of-the-road NVIDIA GPU like the RTX2060.
For toying with autodiff and basic CNNs, CPU works just fine by the way...
That's not quite how the process works. These papers go through multiple (> 2) revisions. At any iteration, there's ample opportunity for updating references. This applies double given how long the Cueva/Wei work has been available (preprint & CCN'17 (?) contribution).
It's certainly an interesting paper, but there's a bit of publication weirdness at play here.
In October '17, Cueva & Wei put out a(n anonymous) paper that recapitulates the core result almost exactly -- that training a recurrent neural network to perform dead reckoning/path integration gives you intermediate units whose place fields strongly resemble grid cells. Critically, this only happens when regularization is applied; Cueva/Wei used noisy inputs and DeepMind implemented 50% stochastic dropout in the intermediate linear layer. There are some superficial differences (generic RNN units vs. LSTM), but at their core these studies are virtually identical. Check it out:
What I don't get -- why doesn't DeepMind acknowledge this result? Sure, the Nature paper was submitted in July '17, but these things go through many revisions. Clearly, DeepMind went a bit further with the whole integrating visual CNNs/grid cells part. Nonetheless: Fig. 1 is the core result, everything from Fig. 2 onwards is nice-to-have but not essential, and I feel like Cueva/Wei got there first.
Ah, well. At least the minor controversy brings in great publicity for the Cueva/Wei paper.
> I've noticed that when I get a blink/flinch response from
> something (usually some sand or a bug hitting my face when
> I'm on the bike), it feels like I blink just a split second
> before the thing hit me.
We know very little about conscious perception or even the locus at which sensory signals are integrated to generate a conscious percept. But it's perfectly possible that delays differ across modalities and that the proprioceptive signal about lid-closing reaches whatever-relevant-area before your visual system catches up.
> I read something once about our perception of audio being
> delayed by ~100ms so that it synchs up with our perception
> of vision
Not an expert on audition, but the brain is really good at generating coherent representations of the physical world across modalities. I wouldn't be surprised if such cross-sensory synchronisation happened in some form.
As someone who works in visual neuroscience, this article's a tough read. Lots of statements that are semi-accurate at best.
1) Eyes don't work like cameras; there's no real "exposure" phase as such (even though there's lots of thresholds). So it's misleading to talk about discrete images that we sample at some fixed frequency. Instead, it's much more helpful to think of photoreceptors and subsequent processing stages as continuous band-pass filters. At some point, high frequencies are simply cut off because the electro-chemistry of the cell can't keep up. For us, that cut-off comes earlier than it does for invertebrates.
2) There's no mechanical interaction between light and photoreceptor. Instead, the transduction cascade of the dipteran eye seems to encompass a mechanical (as opposed to biochemical) step.
3) It's pure conjecture to talk about a fly's slowed down "perception" of the world. The reason why they take off before you get to them is much simpler -- there's a highly optimized reflex that connects eye and flight muscles via the giant fiber (a particularly rapid nerve). We have similar responses, like eye lid closing etc. Additionally, their photoreceptors are sensitive and fast. But there's zero evidence that flies have any sense of continuous time that could be faster than ours.
I only skimmed the notebook, but the code looks fairly inadequate. The pandas portion, for instance, treats the data frame as a dumb array and critically ignores grouping functionality which should offer a tremendous speedup. Moreover, for the exact same task pandas should never be slower than NumPy.
Benchmarks are hard to get right but this one falls way short of saying anything at all about performance penalties incurred by various libraries and abstractions.
We didn't have to reverse-engineer the CPU. Also, CPUs are rationally designed; we have zero guarantees that an evolved brain follows any principles at all. Every brain region may be highly specialised to a single algorithm or task with zero mechanistic overlap. Figuring out deep neural nets is a closer analogy, and at the moment we have very little intuition for how to do even that. Keep in mind that the brain is vastly more complex than even the most advanced ANNs.
On the technical side: The most advanced neural techniques allow for parallel measurement calcium signals in ~10,000 neurones. That's a long way off from complete observation!