Beer and Concurrent HTTP Pipelines(damienradtke.com)
damienradtke.com
Beer and Concurrent HTTP Pipelines
https://damienradtke.com/post/beer-and-http-pipelines/
1 comments
Rather than emoticon-ing away your responsibility in a potential site DoS, perhaps read up on responsible crawling - https://webarchive.jira.com/wiki/display/Heritrix/Responsibl... + http://blog.mischel.com/2011/12/20/writing-a-web-crawler-pol... . You had no need for either unlimited or even 50 parallel crawl tasks. And, if your intent is anything but personal use, you shld prbly (re)read https://www.brewtoad.com/legal since you just got their attention in a pretty big way.