Ask HN: Google web corpus + big data analysis cluster?
1 comments
Some quick questions:
1) How frequently would you crawl?
2) What are you defining as a 'massive scale'? It'll probably grow i'm sure but what is a reasonable range you're working with?
3) Which languages will you be supporting?
1) How frequently would you crawl?
2) What are you defining as a 'massive scale'? It'll probably grow i'm sure but what is a reasonable range you're working with?
3) Which languages will you be supporting?
Obviously walled gardens like Facebook and Twitter would be off limits at the beginning, but if the service gains traction, then it's possible companies would want to be crawled.
I was trying to analyze the common crawl recently, and the process of getting set up is non trivial. This service would allow more people to more easily analyze the web.
Would welcome any feedback.