Http://www.google.com/bot.html = this page no longer available(google.com)1 points·by gstar·18 lat temu·2 commentsgoogle.comHttp://www.google.com/bot.html = this page no longer availablehttp://www.google.com/bot.html2 commentsPost comment[–]gstar·18 lat temureplyCrikey - I thought Google of all places wouldn't stuff up the user-agent bot webpage.Google's user-agent is: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.htmlI mean, everyone has heard of Google - but it's very bad mannered not to have a page up about your crawler.[–]ScottWhigham·18 lat temureplyIts up now
Google's user-agent is: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html
I mean, everyone has heard of Google - but it's very bad mannered not to have a page up about your crawler.