Songkick's robots.txt has a few special rules.(songkick.com)1 points·by nintax·14 yıl önce·2 commentssongkick.comSongkick's robots.txt has a few special rules.http://www.songkick.com/robots.txt1 commentsPost comment[–]dibarra·14 yıl öncereplyIt's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).[–]nintax·14 yıl öncereplyDid you notice the bottom 3 rules? Maybe I should resubmit and change the title...