Webcrawling at Scale with Nokogiri and IronWorker: Part 2(blog.iron.io)
blog.iron.io
Webcrawling at Scale with Nokogiri and IronWorker: Part 2
http://blog.iron.io/2012/09/webcrawling-at-scale-with-nokogiri-and.html
1 comments
Awesome. Love the options.
FWIW - the value prop of IronWorker is never having to deal with servers again -- and only pay for the seconds you're actually crawling. Fire up a million crawlers (workers) and it's auto distributed across large sets of machines behind the scenes (no "spin up tear down" either).
So in essence, it's a completely hosted version of what you described. The power is seen by "just trying it". No software (except for your code), no installations, no servers, etc.
FWIW - the value prop of IronWorker is never having to deal with servers again -- and only pay for the seconds you're actually crawling. Fire up a million crawlers (workers) and it's auto distributed across large sets of machines behind the scenes (no "spin up tear down" either).
So in essence, it's a completely hosted version of what you described. The power is seen by "just trying it". No software (except for your code), no installations, no servers, etc.
How We Built Our 60-Node (Almost) Distributed Web Crawler http://hackerne.ws/item?id=4469911