PiCrawler: A distributed web crawler using PiCloud(github.com)
github.com
PiCrawler: A distributed web crawler using PiCloud
https://github.com/studio-ousia/picrawler
1 comments
I'm rather fond of PiCloud as a platform - it's good to see example projects out there using it.
I am also a fan of using PiCloud as a platform.
Using PiCloud for implementing crawler requires some hacks, so I decided to create this as a separate package.
Using PiCloud for implementing crawler requires some hacks, so I decided to create this as a separate package.
What are the advantages over Rackspace, Amazon, Digital Ocean? Seems they are targeting science and big data apps.
PiCloud is a layer of abstraction above the IaaS providers you've listed. That is to say that it gives you software infrastructure to design workflows and distribute them across the machines provided by an IaaS provider. In some ways it's like Heroku, and in others it's another layer beyond Heroku (App Engine-like?). You architect your work around the PiCloud tools. I think PiCloud currently runs on AWS.
For me? Well, I'm a computational epidemiologist, so targeting science is exactly it's advantage. PiCloud allows the use of AWS resources fairly seamlessly from inside scripts, iPython notebooks, etc.
They're also very responsive to questions, and their package is far, far easier to use for the things I use it for than trying to build something myself via AWS, Rackspace, etc.
Disclosure: They also gave me a $500 research grant.
They're also very responsive to questions, and their package is far, far easier to use for the things I use it for than trying to build something myself via AWS, Rackspace, etc.
Disclosure: They also gave me a $500 research grant.