Hello all. I am one of the paper's coauthors. Thank you for your interest in this work! We hope you enjoy it. Just an FYI, this paper is part of a cluster of five papers released on Tuesday morning, detailed in this blog post: https://eng.uber.com/deep-neuroevolution
I would just unleash evolution. 1 or 2 in the first generation is a toehold, and from there evolution can begin to do its work. You can also try a larger population (e.g. 2000) and let it run for a while.
> Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models. It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another,
Agreed. That is surprising, and also increases the security risks, because I can produce images on my in-house network and then take them out into the world to fool other networks without even having access to the outputs of those networks.
> likely since they share similar training data (?) unclear.
The original Szegedy et al. paper shows that these sort of examples generalize even to networks trained on different subsets of the data (and with different architectures).
Also, you don't necessarily need a lot of predictions from a net to fool it, because (another surprising result) images that fool one net tend to fool others! So I can create fooling images on my in-house net and then take them and fool your net-used-for-some-important-application without getting any feedback from your net. That's very surprising, and does raise serious security concerns.
You say that these images are highly optimized to produce this effect and would not occur by chance, but have you looked at the images in the "fooling" paper?
Some of them are very simple, and DO occur a lot in the world. For example, the alternating yellow and black line pattern would be encountered by a driverless car, and it would think it is seeing a school bus.
Surprisingly, ensembles do not really help. We have tried this and it does not work (the final paper for CVPR 2015 will show these results). Also see the work of Szegedy et al. and Goodfellow et all, which also show that ensembles do not really help.
I agree with most of what you say, but note that nearly all of the images in the paper were generated without the gradient. I.e. all the images produced by evolution did not use the gradient, only the output of the network regarding its prediction confidence. There are some images that use the gradient, but only to show a 3rd class of "fooling images".
PS. It's nice to see our work (both this paper and the NIPS paper on transfer learning) in your class. Thanks for including it. I wish I could have my students take your course!
That's because you know how to use Blender, which most people don't. But the site is not for a situation when you have a specific design in mind, instead it is for when you want to explore the space of possible designs and see suggested designs that you might not think of on your own, such as this: http://endlessforms.com/o/372201/
But I also doubt I could produce all of these shapes in Blender. I would have no idea how to do these, for example:
I noticed a surge in traffic to EndlessForms.com and realized it was coming from this post. Thanks to Joel for mentioning our site[1]! For those of you that don't know what I am talking about, see Joel's comment on the original post where he recommends EndlessForms.com, where you can design objects with evolution and 3D print them. Thanks all for checking it out.
Here are two fun gifs of clever solutions AI came up with:
https://twitter.com/jeffclune/status/974718199722795008
https://twitter.com/jeffclune/status/973605950266331138
Here are some press articles which provide a shorter summary of some fun anecdotes.
popularmechanics.com/technology/robots/a19445627/the-hilarious-and-terrifying-ways-algorithms-have-outsmarted-their-creators/
https://www.newscientist.com/article/8-8-hilarious-ways-ai-h... (paywalled unfortunately)