Google Banishes Nicholas Carr From The Index(roughtype.com)
roughtype.com
Google Banishes Nicholas Carr From The Index
http://www.roughtype.com/archives/2009/06/banished.php
5 comments
There is now a comment on this blog entry purporting to be from someone on the Google web search team claiming that the site was removed because it had been hacked and that Google had attempted to contact the author.
The 'hack' must be very, very minor: I can still visit with Chrome, which supposedly checks Google's 'Safe Browsing' blacklist.
Maybe it's just hidden spam links? In which case, index-removal seems an extreme way to get the owner's attention.
Maybe it's just hidden spam links? In which case, index-removal seems an extreme way to get the owner's attention.
Hi there,
I realize neither this comment nor the one on Nick's blog from my Google colleague Greg are authenticated, so you'll just have to trust me that I'm me :).
We do often remove entire domains from our index when there's evidence of hacking and/or malware... not to be mean, but rather to protect both visitors to that site and the site owner (from angry visitors). We can't be sure that the hacked links aren't or won't soon become particularly malicious (e.g., phishing sites), nor can we be certain that the server hasn't been or won't soon be more substantially compromised (with drive-by malware installs, etc.), so we fall on the side of caution.
We do make a good faith effort to contact the webmasters affected... - Through e-mail addresses we believe to belong to that webmaster. - Via our free Webmaster Tools service, where webmasters can sign up and opt to get e-mails from Google on just such occasions as this.
For more info, you can check out our most recent blog post on this topic: http://googlewebmastercentral.blogspot.com/2009/02/best-prac...
Hope this helps! Believe me, we want good content back in our index as soon as possible, too :)
I realize neither this comment nor the one on Nick's blog from my Google colleague Greg are authenticated, so you'll just have to trust me that I'm me :).
We do often remove entire domains from our index when there's evidence of hacking and/or malware... not to be mean, but rather to protect both visitors to that site and the site owner (from angry visitors). We can't be sure that the hacked links aren't or won't soon become particularly malicious (e.g., phishing sites), nor can we be certain that the server hasn't been or won't soon be more substantially compromised (with drive-by malware installs, etc.), so we fall on the side of caution.
We do make a good faith effort to contact the webmasters affected... - Through e-mail addresses we believe to belong to that webmaster. - Via our free Webmaster Tools service, where webmasters can sign up and opt to get e-mails from Google on just such occasions as this.
For more info, you can check out our most recent blog post on this topic: http://googlewebmastercentral.blogspot.com/2009/02/best-prac...
Hope this helps! Believe me, we want good content back in our index as soon as possible, too :)
Thanks for the response and link.
The "could get worse" standard is a slippery slope, unless the evidence suggests a very specific serious compromise or foreign active content. (Exiling a site because a few evil links appear in its comment threads, for example, would put almost every site at risk of removal. Indeed, that standard risks gaming by a site's rivals, using comment forms rather than 'hacks'.)
Also, looking at Carr's last comment, even though he's removed the bad links, he doesn't know what compromise allowed them to appear. So his site is still in the "can't be sure... won't soon become particularly malicious... [or] substantially compromised" category that justified its initial removal.
Your deep and sensitive checks for problems are definitely a public service... as long as the standards are clear and communications effective. Maybe notification should occur via a site's own comment functionality in addition to email? No details for third parties to exploit -- just a "please check Google Webmaster Tools" note.
The "could get worse" standard is a slippery slope, unless the evidence suggests a very specific serious compromise or foreign active content. (Exiling a site because a few evil links appear in its comment threads, for example, would put almost every site at risk of removal. Indeed, that standard risks gaming by a site's rivals, using comment forms rather than 'hacks'.)
Also, looking at Carr's last comment, even though he's removed the bad links, he doesn't know what compromise allowed them to appear. So his site is still in the "can't be sure... won't soon become particularly malicious... [or] substantially compromised" category that justified its initial removal.
Your deep and sensitive checks for problems are definitely a public service... as long as the standards are clear and communications effective. Maybe notification should occur via a site's own comment functionality in addition to email? No details for third parties to exploit -- just a "please check Google Webmaster Tools" note.
Hey Gojomo, appreciate the input!
I definitely hear you on the slippery-slope issue, and perhaps I could have worded my initial note more carefully. We have reasonably conservative heuristics in place to assess whether a site has been actually hacked vs. just someone adding, say, a link to a nasty site in a comment. Also, in our experience, once a site or server's been compromised, there's a good chance that, well, where there's smoke, there's fire.
Regarding giving more information... I believe we do generally show specific examples re: hacking and malware via our Webmaster Tools, though I can't say for sure what we presented in Nick's case. Generally, we've seen that webmasters are able to catch the root vulnerability (via their own unpatched software or via their server), which prevents the same problem from recurring. (and speaking of unpatched software... from my own spot checks, that seems to be one of the most major culprits!)
Lastly, re: your suggestion about notification on the site's comment functionality itself... that's a fascinating idea! I think that it'd be a bit challenging to implement, given the diverse array of registration requirements, captchas, etc. (our bots are smart, but not as smart as humans!), but I know that our team is looking into more ways that we can let webmasters know of issues more effectively. Thanks for the suggestion!
I definitely hear you on the slippery-slope issue, and perhaps I could have worded my initial note more carefully. We have reasonably conservative heuristics in place to assess whether a site has been actually hacked vs. just someone adding, say, a link to a nasty site in a comment. Also, in our experience, once a site or server's been compromised, there's a good chance that, well, where there's smoke, there's fire.
Regarding giving more information... I believe we do generally show specific examples re: hacking and malware via our Webmaster Tools, though I can't say for sure what we presented in Nick's case. Generally, we've seen that webmasters are able to catch the root vulnerability (via their own unpatched software or via their server), which prevents the same problem from recurring. (and speaking of unpatched software... from my own spot checks, that seems to be one of the most major culprits!)
Lastly, re: your suggestion about notification on the site's comment functionality itself... that's a fascinating idea! I think that it'd be a bit challenging to implement, given the diverse array of registration requirements, captchas, etc. (our bots are smart, but not as smart as humans!), but I know that our team is looking into more ways that we can let webmasters know of issues more effectively. Thanks for the suggestion!
The comments on the article explain why, Google detected that the site had been hacked and had email the author of the site.
Curious, the search http://www.google.com/search?q=roughtype.com
Brings up all sorts of URL's that contain the string roughtype.com but are not the site itself.
It's frightening, especially since we have no way of knowing what else is being filtered out and for whom.
I would tend to give the big G the benefit of the doubt; it's a mistake or he got on the wrong side of the algorithm, or his site was infected with malware. (that last one does happen see http://blog.oddhead.com/2009/06/22/un-hacking-my-blog/ )
But still, it is the reputational equivalent of logging in to your brokerage account to find out it's empty.
Brings up all sorts of URL's that contain the string roughtype.com but are not the site itself.
It's frightening, especially since we have no way of knowing what else is being filtered out and for whom.
I would tend to give the big G the benefit of the doubt; it's a mistake or he got on the wrong side of the algorithm, or his site was infected with malware. (that last one does happen see http://blog.oddhead.com/2009/06/22/un-hacking-my-blog/ )
But still, it is the reputational equivalent of logging in to your brokerage account to find out it's empty.
Why is that curious? You're searching for mentions of his domain, not content on his domain. To search a site, you preface with site:. http://www.google.com/search?q=site:roughtype.com doesn't contain any content (because he had spam on his domain and he'll make it back in the index in a couple of days after he removed it.)
No way to know for sure, but it most likely occurred because of this: http://www.roughtype.com/archives/2009/05/for_whom_the_go.ph...
Why do you assert that it "most likely occurred" because of this?
Do you have any data?
Do you have any data?
I doubt they would be that petty, if it became a common thing for Google to apply the banhammer for mild critiques of their business practices that would cause irreparable damage to the brand.
We would all be trying meta-search engines that compared different result sets. We would be back in the late '90's trying to figure out which search engines were reliable and honest. (anyone remember Dogpile?)
Except that now it's all incumbent monopolies and high barriers to entry.
If nothing else this should give a boost to small outfits like DuckDuckGo http://duckduckgo.com/?q=site%3Aroughtype.com
We would all be trying meta-search engines that compared different result sets. We would be back in the late '90's trying to figure out which search engines were reliable and honest. (anyone remember Dogpile?)
Except that now it's all incumbent monopolies and high barriers to entry.
If nothing else this should give a boost to small outfits like DuckDuckGo http://duckduckgo.com/?q=site%3Aroughtype.com
Wow! This is crazy. A similar thing happened to my friend Matt Harrison a while back. Here are the relevant links:
- http://panela.blog-city.com/burninated_by_google.htm
- http://panela.blog-city.com/back_in_google.htm
- http://panela.blog-city.com/google_hates_me.htm
As Matt explains, he was re-added to the index with no explanation. Perhaps Nick Carr will have a similar redemption.
As Matt explains, he was re-added to the index with no explanation. Perhaps Nick Carr will have a similar redemption.
As indeed he has (see last comment on OP link).
[deleted]