Google gets preferential permissions in robots.txt(knuckleheads.club)
knuckleheads.club
Google gets preferential permissions in robots.txt
https://knuckleheads.club/introduction/
11 comments
Makes sense until you realize you'd have to prohibit everyone else's own crawling as well (can't single out and put Google at a disadvantage) and then you'd just have turned Google's de facto-monopoly into an actual, government-mandated crawling monopoly.
I'd rather make http://commoncrawl.org more current, accessible and more commonly used so website publishers can see a benefit in actively supporting it to lighten the load on their servers.
I'd rather make http://commoncrawl.org more current, accessible and more commonly used so website publishers can see a benefit in actively supporting it to lighten the load on their servers.
It's a natural monopoly. It has already de facto prohibited everyone else's own crawling, as the linked article demonstrates.
When it had a monopoly, AT&T was forbidden from selling software.
When it had a monopoly, AT&T was forbidden from selling software.
I disagree on the "natural" part. Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years, some websites supporting only Netscape and MSIE, a direct consequence of Google's current market share and might change once there is a good reason (like DDG growing into a significant player).
If a collection like commoncrawl with bulk downloads was more useful and thus used more often, even Google would have a good reason to use it.
If a collection like commoncrawl with bulk downloads was more useful and thus used more often, even Google would have a good reason to use it.
> Robots.txt that put other search engines at a disadvantage aren't the norm, they're, just like in the early years
It's not just robots.txt, it's also cloudflare and IP-based throttling. And it is very, very commonplace: http://gigablast.com/blog.html
It's not just robots.txt, it's also cloudflare and IP-based throttling. And it is very, very commonplace: http://gigablast.com/blog.html
Not commonplace https://news.ycombinator.com/item?id=25373909
What if i don't want Google to crawl my website but i'm ok if others do?
As it currently is you can't stop them from using an undeclared IP range to crawl you.
This doesn't change that.
"Everybody except X is allowed" rules have always been "on your honor" type restrictions. "X" can always claim to be Joe Rando.
This doesn't change that.
"Everybody except X is allowed" rules have always been "on your honor" type restrictions. "X" can always claim to be Joe Rando.
Anyone got a TL:DR? The crux of their argument isn't even on the first page...
They found a bunch of robots.txt files on sites like census.gov that blocked all robots except a few known ones, especially GoogleBot
Everybody (including Google) pays the same price per query/byte/whatever. CrawlCo gets ownership of the crawling IP range and is prohibited from entering any other market. GoogleCo is prohibited from doing its own crawling, but can ask (perhaps require) CrawlCo to add new crawl products which are then available to everybody.
Basically force Google to do with their crawler what Amazon voluntarily did with their datacenters.