Ask HN: The best way store thousands of Images on a server
8 comments
I recommend gatling: http://www.fefe.de/gatling/
It's the fastest httpd I know of.
You might also want to try thttpd: http://www.acme.com/software/thttpd/ I heard all the porn sites use it to host their images.
With only the http server for the images running it should work quite well to rely on the filesystem cache.
You might also want to try thttpd: http://www.acme.com/software/thttpd/ I heard all the porn sites use it to host their images.
With only the http server for the images running it should work quite well to rely on the filesystem cache.
MogileFS might be what you're looking for:
http://www.danga.com/mogilefs/
with an nginx or lighttpd frontend.
The documentation is a little sparse though the mailing list and irc are useful for help.
Also it would somewhat depend on your traffic profile. If you've got thousands of random reads (e.g. thumbnails with dozens per page, ssds might be better; specifically the Intel X-25).
Otherwise MogileFS with an array of fast drives would probably do the job. Mogile can also do load balancing based on drive I/O because each image is replicated across different files/machines.
EDIT: Make sure you've got browser caching set up correctly (though admittedly that's a tiny bit harder with Mogile)
If you can use a expiry time that's far in the future. Also don't send a Last-Modified nor an ETag Header. Only send Expires.
http://www.danga.com/mogilefs/
with an nginx or lighttpd frontend.
The documentation is a little sparse though the mailing list and irc are useful for help.
Also it would somewhat depend on your traffic profile. If you've got thousands of random reads (e.g. thumbnails with dozens per page, ssds might be better; specifically the Intel X-25).
Otherwise MogileFS with an array of fast drives would probably do the job. Mogile can also do load balancing based on drive I/O because each image is replicated across different files/machines.
EDIT: Make sure you've got browser caching set up correctly (though admittedly that's a tiny bit harder with Mogile)
If you can use a expiry time that's far in the future. Also don't send a Last-Modified nor an ETag Header. Only send Expires.
Check out this Stanford presentation
Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos
http://www.flowgram.com/p/2qi3k8eicrfgkv/
Facebook - Needle in a Haystack: Efficient Storage of Billions of Photos
http://www.flowgram.com/p/2qi3k8eicrfgkv/
How about Publicfile? - http://cr.yp.to/publicfile.html
Super duper secure and very, very quick, especially for smaller files.
I'd try and run some tests against it, thttpd and your existing apache setup.
Super duper secure and very, very quick, especially for smaller files.
I'd try and run some tests against it, thttpd and your existing apache setup.
A smaller web server like lighttpd or cherokee may be a better choice if all you are serving is images.
dont forget nginx
I have Apache 2 on Linux and nothing else. I don't know if it is the best solution, but I think is not.
What system (http server, cache, or anything) would you recommend me?