Bottom line is that if you don't trust Google & use their tools (GWT) then you can end up in sticky situations like this one.
My experience with the Crawl Rate feature via GWT is that they do honour it pretty strictly, but for large sites Gbot can cause a lot of extra load even if pages are static.
A good CDN and stateless cache server will help but for sites as large as HN every request adds up!
My experience with the Crawl Rate feature via GWT is that they do honour it pretty strictly, but for large sites Gbot can cause a lot of extra load even if pages are static.
A good CDN and stateless cache server will help but for sites as large as HN every request adds up!