What's the hidden secret of the review for ASIN B0069IY63Y?(amazon.com)

1 pointsby pdknsk14년 전3 comments

http://www.amazon.com/robots.txt

2 comments

I don't get it. The link points to www.amazon.com/robots.txt.

wimplash14년 전

if you look in the file you'll see a Disallow entry for /product-reviews/B0069IY63Y.

wimplash14년 전

honeypot entry to ensure that crawlers are complying with robots.txt?