Googlebot snarfing URLs from the ether?
2 comments
Did you access that URL with Google Chrome mere seconds before googlebot miraculously found it?
Or does the browser your using have instance results from the address bar and the search engine is Google?
Yes, though I've tried testing for this, but I only see requests from my browser (Opera) for incomplete URLs.
No, I'm using Opera (see the log). It's possible Opera is sending such info as well (it uses Google for autocompletion), but I've tried reproducing this and I don't see evidence of it. (I do see autocompletion requests, but never for complete URLs.)
That is surprising and somewhat troubling.
Wait, are you saying the Google Chrome will send URLs back to Google for indexing? I did not know this
I'm not saying it as a statement of fact. But rather as a plausible explanation for what OP experienced.
But if I were google I would definitely give people free tools with which they could perform, with actual human intelligence, tasks I had found difficult.
But if I were google I would definitely give people free tools with which they could perform, with actual human intelligence, tasks I had found difficult.
Two days later, I get an e-mail from Google's Webmaster tools (which I have configured on that server) that my site is down. Indeed, it needs another reboot, as if someone accessed that private URL. But surely no bot could know that URL exists, right? Checking my server logs I find this (1.2.3.4 is my IP address):
That is, mere seconds after I had first accessed a URL the world had never seen before, Googlebot decided to crawl my site (as indicated by the robots.txt hit), and then access the URL that was not linked anywhere.
My only explanation is that somewhere between my home computer and my (off-site) server, a Google-affiliated network appliance sniffed my initial GET and sent the URL to Googlebot to be indexed within 30 seconds.
Has anyone else experienced this?
(Yes I know security through obscurity, should use robots.txt, yadda yadda.)