Web-scale crawling at cable modem speedsblog.metadatalabs.com8 points·by jmischel·16 anni fa·0 comments
jmischel·15 anni fa·discussI ran into this with my crawler, too. See http://blog.mischel.com/2008/08/08/hey-you-deleted-my-files/