All the steps needed to create deep dream animations on EC2(github.com)
github.com
All the steps needed to create deep dream animations on EC2
https://github.com/JohnMount/CaffeECSExample
7 comments
For better or for worse, with shell, the line between "a list of instructions to be typed in" and "shell script" is very, very thin.
Toss a
(set -e will make the script stop when something returns with an error exit code. Despite temptation, given the size of this script I do advise doing your best to work with that... this is just the sort of script that if you let it blunder ahead blindly, when something doesn't work you'll need an hour just to figure out what didn't work, let alone fix it....)
Toss a
#!/usr/bin/bash
set -e
(set -e will make the script stop when something returns with an error exit code. Despite temptation, given the size of this script I do advise doing your best to work with that... this is just the sort of script that if you let it blunder ahead blindly, when something doesn't work you'll need an hour just to figure out what didn't work, let alone fix it....)
Some of the steps require user input. Is the "set -e" for dealing with that? Also at least one reboot is needed, the script depends on a lot of external sources being up, and I have seen different behavior on the kernel upgrade steps.
No, see edit. But there are ways to do user input in shell, see Google. (By which I mean I literally do not personally know the best answers and that's what I'd be doing anyhow if I tried to explain here.)
Thanks for updating and organizing all of this. I got a working EC2 instance with CUDA and CUDNN working yesterday but it took some time. Having all of this is one place is excellent.
These steps are cool and very detailed, also check out https://github.com/VISIONAI/clouddream which is containerized so you can try it on local machine as well as on EC2.
Thanks, that might be the better option for a lot of people. I'll add that to the readme.
That's pretty insane. It makes me really appreciate that I get to use package managers and Puppet/Vagrant scripts for most of the stuff I program.
Yeah, I don't normally "do my own infrastructure" (I know, "it shows"). Usually the client or vendor has supplied the build. And I didn't want to commit to a package manager early in the exercise (in case it turned out to be incompatible with something one of the dependent projects did). So I just tried to start from the Caffe instructions and then fix things that broke.
The main components turned out to be: NVIDIA/CUDA, python, Caffe, ffmpeg.
I was very surprised how many steps ffmpeg took.
The main components turned out to be: NVIDIA/CUDA, python, Caffe, ffmpeg.
I was very surprised how many steps ffmpeg took.
Would you be willing to publish an AMI of the completed image? Seems like that'd make it really easy to run without all of the package management work.
Yes, I have a running instance up now in USWest. If it is free and somebody could shoot me instructions I'll make it an AMI (after lunch, on the way out the door right now).
Actually I started looking into it. I don't think I am going to make the AMI.
First I am not sure if the NVIDIA driver licenses do or do not allow this. Second I would have to tear down my current run and pay S3 charges to distribute the AMI. Third the instructions are long and ugly ( http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-... ) even after seeing the setup steps. Finally as people have mentioned in other comments there is a good to go Docker image for people to also play with.
Sorry about that.
First I am not sure if the NVIDIA driver licenses do or do not allow this. Second I would have to tear down my current run and pay S3 charges to distribute the AMI. Third the instructions are long and ugly ( http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-... ) even after seeing the setup steps. Finally as people have mentioned in other comments there is a good to go Docker image for people to also play with.
Sorry about that.
No problem.
For what it's worth, I've had luck just using the AWS EC2 web console, and choosing "Create Image" and "No reboot" from the instance's context menu. That spits out an EBS snapshot and an EBS-backed AMI based on the snapshot, which is probably what you were after. I went through the instance store image creation steps a long time ago for a project that deployed on custom AMIs and it was pretty convoluted; we ended up going EBS-backed and skipping the instance store altogether.
For what it's worth, I've had luck just using the AWS EC2 web console, and choosing "Create Image" and "No reboot" from the instance's context menu. That spits out an EBS snapshot and an EBS-backed AMI based on the snapshot, which is probably what you were after. I went through the instance store image creation steps a long time ago for a project that deployed on custom AMIs and it was pretty convoluted; we ended up going EBS-backed and skipping the instance store altogether.
I now list other people's docker and AMI instances in my README, that should help.
This is really cool, but honestly, I'm getting a bit tired of seeing all those "eyes". I want to feed it my own training image set. Is that possible?
Alternatively, is there a way to specify I don't want to enhance on certain outputs, like "dogs" and "faces"?
Alternatively, is there a way to specify I don't want to enhance on certain outputs, like "dogs" and "faces"?
Thanks, the Caffe on the PYTHONPATH was the bit I needed.
Or... Just use deepdreamer.io
That is definitely the more convenient option for running the defaults, but I think this is for a different usecase. If someone wanted faster results and full access to edit the python code and caffe models, this would be the better option.
It's a bit better than the defaults because it generates many at once. But ya if you really wanna hack on it...
tacos(3)
To do this on a fresh Ubuntu EC2 g instance there are a lot of steps- but I have tested them and put them all in one place (with links to the original sources and guides). I have CUDA up but not CUDNN as I haven't found how to legitimately download CUDNN without registering on the NVIDIA website.
Again: credit to the actual creators and all the original guide authors.