Processing photos on demand with Photon(blog.1000memories.com)
blog.1000memories.com
Processing photos on demand with Photon
http://blog.1000memories.com/169-photon
16 comments
How do you communicate with Photon, is it over HTTP? (I know nothing of this "JAX-RS" you speak of). I was thinking of building something like this for my photo warping app Trollaroid -- currently all the processing is done on client devices, but it'd be much more sensible to do it in the cloud. How might hooking up my API endpoint to an array of Java image processors over HTTP compare to say, using ZeroMQ as the glue?
Our webapp doesn't directly communicate with photon at all. The webapp sticks the photo in a certain bucket on S3 with, and then includes the path to that photo in the photon url that gets passed to the client.
On the read path, HTTP is really nice because browsers, caches, and CDNs all speak HTTP.
On the read path, HTTP is really nice because browsers, caches, and CDNs all speak HTTP.
The name "Photon" is used by at least three other opensource projects:
* A library of utility code by Alvaro Carrasco [1]
* A Mongrel2-PHP web apps framework [2] (which is what I was expecting the linked page would be about)
* A framework for Silverlight applications [3]
I know it's hard to choose names, but really, 10 seconds with google should have suggested this particular name is already a bit crowded.
[1] http://www.photonframework.com/wiki/index.php/Main_Page [2] http://www.photon-project.com/ [3] http://photon.codeplex.com/
* A library of utility code by Alvaro Carrasco [1]
* A Mongrel2-PHP web apps framework [2] (which is what I was expecting the linked page would be about)
* A framework for Silverlight applications [3]
I know it's hard to choose names, but really, 10 seconds with google should have suggested this particular name is already a bit crowded.
[1] http://www.photonframework.com/wiki/index.php/Main_Page [2] http://www.photon-project.com/ [3] http://photon.codeplex.com/
How does this compare to something like Dragonfly (https://github.com/markevans/dragonfly/)? 1000memories's issue was with the time it took to generate 12 different versions, not the speed of Imagemagick itself. So I'm not quite sure why they needed to write their own extension. Even Carrierwave (plus caching of course) can be setup this way (https://gist.github.com/1541912).
Dragonfly is quite nice, but we opted not to use it for a handful of reasons: ImageMagick itself is a actually a good bit slower for basic resizing/cropping/rotating than the library we're using in Photon; Photon is ~100 SLOC, Dragonfly is ~3500; Photos are the core of our business, so a generic solution is not likely to work 100% how we need it to (we have various legacy concerns that probably would've require more code to monkey patch Dragonfly than to just write it into Photon); Java is much better than Ruby for CPU and RAM bound tasks that need to execute in parallel.
All that said, Dragonfly is a great choice for most people.
All that said, Dragonfly is a great choice for most people.
I would be interested in a response to this as well....
Thanks for open sourcing this 1000mem. What is average request latency for a 2mb photo? We tried to do something similar with python, but we found there's a 300ms latency between s3 and ec2. It takes quite a few seconds to transfer from s3 to ec2, resize it, and transfer to client. You mentioned using caching to speed things up, but what about the first request?
We do two layers of caching (CDN in front of Varnish in front of photon). The latency is annoying, but it's no worse than serving the files directly off of S3 (as we were doing previously).
In an ideal world, Photon would be running on boxes that store the files themselves, and some sort of intelligent load balancer would make sure that the requests get directed to the correct box.
In an ideal world, Photon would be running on boxes that store the files themselves, and some sort of intelligent load balancer would make sure that the requests get directed to the correct box.
If you're interested in other examples of this sort of solution, this is how Shopify has tackled this problem: https://github.com/tobi/imagery
It looks like tobi also has played around with porting it to go as well: https://github.com/tobi/imagery-go
It looks like tobi also has played around with porting it to go as well: https://github.com/tobi/imagery-go
I tried to write an interface to ImageMagick's MagickWand in Golang as well to solve the same problem. You can find it here: https://github.com/dqminh/mage
Awesome to see something like this open sourced. Definitely going to check this out. For those who don't have the resources available to manage this kind of infrastructure however, you may wish to look at Cloudinary (http://cloudinary.com/).
For low volume sites, photon works well enough on a single Heroku dyno (especially if you put a cache in front of it).
I'm sure it works great. But it's one other piece of infrastructure to deploy, monitor, and maintain. Depending on what stage your at and whether or not photos are core to your business, I think (at least temporarily) a hosted solution is better for some.
[deleted]
need to use progressive jpeg if picture is large: http://photon-example.herokuapp.com/lily199iu;w=2000
I also wish it did that, as well as stripping meta data and exif rotations. Or perhaps this is all automatic?