Web-scale crawling at cable modem speedsblog.metadatalabs.com8 points·by jmischel·hace 16 años·0 comments
jmischel·hace 15 años·discussI ran into this with my crawler, too. See http://blog.mischel.com/2008/08/08/hey-you-deleted-my-files/