I disagree with this article in so many ways. I have a CS degree and I work with many people who don’t, and they are just as good as (or even better than) me at all the points raised by the article.
They meet deadlines, they are incredibly good at communication and collaboration and they have pretty good networking. Most of these traits come from the fact that they needed to develop them in order to succeed in learning by themselves.
It is a pretty limited view of the world to think that only college can bring you this. Immersing yourself in a coding bootcamp for some people means leaving the jobs they need to survive in order to have a better job in the future. I can’t imagine how being on college can teach more about meeting deadlines, teamwork, communication and perseverance than that.
I wish this article provided more facts to back its beliefs up.
Crawl-delay is not in the standard robots.txt protocol, and according to Wikipedia, some bots have different interpretations for this value. That's why maybe many websites don't even bother defining the rate limits in robots.txt.
Scrapy is asynchronous, but it provides many settings that you can use to avoid DDoS a website, such as limiting the amount of simultaneous requests for each domain or IP address.
And yes, crawling politely requires a bit of effort from both ends: the crawler and the website.
This is such an inspiring story! I guess a lot of people today go the opposite way, renting or buying places where they know there is a decent internet access. I'm guilty about it. :)
Hey, of course. We are glad that you are interested in testing Kumo.
Please email us (help at scrapinghub . com) your user id, organization ids and the project ids you want to migrate to Kumo. Then we'll get back to you, giving early access to Kumo beta and documentation.
Just keep in mind that it is still an experimental platform.
Hey, I'm the author of this post. Feel free to ask any questions or to suggest topics for the next month's post on the "Scrapy tips from the pros" series. :)
- developers who want to develop some data-based product (a travel agency website, who finds the best deals from airline companies);
- lawyers can use it to structure the data from Judgments and Laws (so that they are able to query the data for things like: which judges have interpreted this law in their judgments) (more on this: http://blog.scrapinghub.com/2016/01/13/vizlegal-rise-of-mach...)
- (data-)journalists who work on investigative data-based articles (they use it to gather the data to build visualizations, infographics, and also to support their arguments).
- real state agencies can use it to grab the prices of their competitors, or to get a map of what people are selling, what are the areas where there is more demand.
- large companies that want to track their online reputation can scrape forums, blogs, etc, for further analysis.
- online retailers that want to keep their prices balanced with their competitors can scrape the competitors websites collecting prices from them.
It’s a collection of snippets to explain behaviors that may be considered unexpected.