Darts: Differentiable architecture search for convolutional, recurrent networks(github.com)
github.com
Darts: Differentiable architecture search for convolutional, recurrent networks
https://github.com/quark0/darts
3 comments
Why do you think that throwing compute at the problem is absurd? Arguably that's the approach that produced the resurgence of interest in CNNs.
Yes I think the whole point of Zoph and Le's research is to empirically demonstrate the desired trade off of compute cycles vs feature engineering
I'm not the original commenter, but I think it's fair to say that throwing compute at the problem is absurd from an academic point of view (where the goal is understanding), but not from a commercial point of view. Although there's still some academic value in throwing compute at a problem, just to see what's possible.
Can you link the paper of Zoph,Le that did the “1000 GPUs” thing? Thanks! I tried searching for it but apparently there was quite a few papers on parameter search.
Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables).
1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268
1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268
Cool, was thinking about this subject earlier today.
Abstract:
>This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.
Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578
Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...
Abstract:
>This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.
Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578
Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...
Not to get existential here, but the analogy to human neural networks is compelling. The algorithm efficiently redesigns itself to fit the data, just like neural synapses reconnect one another as knowledge is learned, to optimize the storage and retrieval of the information.
And a huge +1 for the inclusion of random architectures in the results.