Garbage In, Garbage Out: Why Scraping Doesn't Work for Local Search(blog.nelso.com)
blog.nelso.com
Garbage In, Garbage Out: Why Scraping Doesn't Work for Local Search
http://blog.nelso.com/2009/07/garbage-in-garbage-out-why-scraping.html
4 comments
"We've tried scraping ourselves in the past (yes, it's perfectly legal),"
Yes.
Facts can't be legally protected.
A compilation is protected only as a completed work. You can't, for example, photocopy the yellow pages and resell it. But you could copy each and every phone number from the yellow pages and use those numbers in your competing directory.
A compilation is protected only as a completed work. You can't, for example, photocopy the yellow pages and resell it. But you could copy each and every phone number from the yellow pages and use those numbers in your competing directory.
Is that true in all cases? There's dispute about who owns sports statistics because the leagues are trying to claim they have licensing rights to them.
These would be the same people that claim to have rights over who can let who watch their broadcasts, no? I don't see how anyone can have licensing rights to stats regarding events.
So long as it is presented to your web browser, you can also read it with a spider (wget, etc). However, your question suggests that what you do with the data might fall under some sort of copyright if you are going to republish it. I have a book "Spidering Hacks" that discusses this, but I am unable to locate my copy.
One additional point that the book makes is that one should keep in mind is your robot showing up in the logs of places you search. You don't want to overload sites, or get on the wrong section of the site admin's radar.
One additional point that the book makes is that one should keep in mind is your robot showing up in the logs of places you search. You don't want to overload sites, or get on the wrong section of the site admin's radar.
They are only keeping a small amount of information (name, type of store, location ect) which links back to the original site I'd imagine?
In that sense it is legal.
"For example, a recipe is a process, and not copyrightable, but the words used to describe it are; see Publications International v Meredith Corp. (1996).[2] Therefore, you can rewrite a recipe in your own words and publish it without infringing copyrights. But if you rewrote every recipe from a particular cookbook, you might still be found to have infringed the author's copyright in the choice of recipes and their "coordination" and "presentation", even if you used different words"
So, facts are not copyrighted, but the way facts are presented are copyrighted.
I still find it hard to believe that what this person had been doing is legal. What is the difference between the scraping he does and copying the phone directory and selling it on. In fact that is not a fair comparision since the wikepidia article suggests that the fone directory was legaly obliged to list these businesses while an individual or google and yahoo are not legaly obliged to enter every business in their directory. In this regard then the choice of information that these directories have made makes their directories copyrighted perhaps.
So, facts are not copyrighted, but the way facts are presented are copyrighted.
I still find it hard to believe that what this person had been doing is legal. What is the difference between the scraping he does and copying the phone directory and selling it on. In fact that is not a fair comparision since the wikepidia article suggests that the fone directory was legaly obliged to list these businesses while an individual or google and yahoo are not legaly obliged to enter every business in their directory. In this regard then the choice of information that these directories have made makes their directories copyrighted perhaps.
There will always be garbage in. you're algorithms have to overcome this for the most part. Some things have to be manually dealt with and some things could be manually dealt with, but it's impossible to manually verify tens of millions of local listings.
That was the point of the post. Without some manual oversight of every new listing, too many errors will creep in to make the database truly useful.
Local search is not like general text search, where one expects that many of the results returned are not relevant, are wrong, are spam, etc. and can be ignored. Having the incorrect address or phone number for a business on a local search site is a major irritant to the user, and will quickly erode one's user base.
Local search is not like general text search, where one expects that many of the results returned are not relevant, are wrong, are spam, etc. and can be ignored. Having the incorrect address or phone number for a business on a local search site is a major irritant to the user, and will quickly erode one's user base.
I think you missed my point. Manual oversight of tens of millions of listings is not remotely practical. The article points out bad data in google, and it hasn't eroded their user base ;) Don't you think there is a reason google hasn't manually reviewed each listing?
The article points out bad data in google, and it hasn't eroded their user base ;)
I don't trust Google at all for local information. I use social networks and ask my local friends rather than Google when I'm looking for a restaurant or other business in my town.
I don't trust Google at all for local information. I use social networks and ask my local friends rather than Google when I'm looking for a restaurant or other business in my town.
apples and oranges
It looks like it still needs a lot of work. As a quick test I looked for Sports Bars in London (via their categories) and it returned an Antique Shop in Westerham. I then tried editing the record to remove irrelevant categories and got a server error.
The only cities we have good data for are Prague, Copenhagen, and lower Manhattan. Try http://www.nelso.com/cz/prague/
The courts found that it isn't possible to copyright facts, and that's all we were scraping - things like addresses, business name, and phone number. We weren't even scraping things like business category, because something as simple as putting a restaurant in the category "Fine Dining" might be considered a judgment call and therefore value-add by the original site.
And think of what would have happened if the court had found otherwise (i.e. had found that lists of facts could be copyrighted). If you opened a store, and I was the first one to put your address and phone number on-line, no one else could ever include your address or phone number on their site. Even if you created a website for your own business after I published your address, you wouldn't be able to include it on your site, because you'd violate my copyright.
I can't see how the Supreme Court could have ruled any other way.