Web-scale crawling at cable modem speedsblog.metadatalabs.com8 points·by jmischel·há 16 anos·0 comments
jmischel·há 15 anos·discussI ran into this with my crawler, too. See http://blog.mischel.com/2008/08/08/hey-you-deleted-my-files/