Ask HN: Examples of interesting robots.txt

1 pointsby r4vik15 anni fa2 comments

I'm sitting here going through some of our competitors robots.txt files wondering if they were doing anything interesting or innovative but they all seem pretty run of the mill. HN can you share some robots.txts that do something unusual or innovative?

1 comments

balducci15 anni fa

How "innovative" do you expect robots.txt to be?

r4vik15 anni fa

Things like rate-limiting google by serving different disallows on every request, or tuning sitemaps for different crawlers.