Crawl an entire domain in one line(blog.zillabyte.com)
blog.zillabyte.com
Crawl an entire domain in one line
http://blog.zillabyte.com/2014/08/04/crawl-entire-domain-one-line/
6 comments
This post was killed by user flags.
I've just flagged this due to the mechanical '10/10 great job, would crawl again' comments popping up, though I'm not sure this is best practice as the comments are suspect, not necessarily the linked story.
Is there a better approach to use?
Is there a better approach to use?
Actually the comments are a good mix of positive & negative.
Why pay for this when there are OSS/Libre tools that do this for the proper cost of $0?
The example e.g. uses Anemone (https://github.com/chriskite/anemone):
https://github.com/zillabyte/domain_crawl/blob/a22e1fe338e28...
https://github.com/zillabyte/domain_crawl/blob/a22e1fe338e28...
Web scraping is just one example of what Zillabyte can do. The open source libraries out there can be a good solution, but our cloud platform lets you scrape the web at scale. That's the main value we can provide in terms of web scraping.
We also support analyzing internal data such as logs. We also provide a pre-scraped web corpus for our users to mine.
We also support analyzing internal data such as logs. We also provide a pre-scraped web corpus for our users to mine.
Here is another effective one-liner:
wget --recursive --html-extension --page-requisites --convert-links www.randomwebsite.com
Zillabyte allows even fairly non-technical types like me to extract information from the largest repository in history -- the web. I'm a non-traditional user of the tool, but have found it's application for social science researchers to be one of adding an extra dimension to my information set. Thank you Zillabyte.
Zillabyte radically simplifies the process of setting up a web crawler. More and more data-minded developers and marketers are looking to create custom data sets from the open web and these kinds of components combined with ZBs infrastructure will make it much faster and easier to roll your own big data. One to watch.