Ask HN: Who uses web scraping?
2 comments
Yes I had done web scrapping commercially, sometime back when I had just left college. I had to scrap website(s) for getting data about all pharma products available in india. So basically market research. But as soon as I realized this was against TOS of the website, i deleted all data and communicated this to client. I had to discard the contract and return even the previous payments.
Although I am happy that I did. :)
Eventbrite used scraping in the early days. They first argued it was a publicly available website. "Public" in the sense that the site was browse-able without restriction.
The plaintiff responded that it had anti-scraping language in its user agreement. The court said "the Terms of Use are not displayed on the website in any way in which a reasonable user could be expected to notice them."
The plaintiff responded that it had anti-scraping language in its user agreement. The court said "the Terms of Use are not displayed on the website in any way in which a reasonable user could be expected to notice them."
[1] - Yes
[2] - Lead generation, selling data, consumer + market research
[3] - I think it is always a gray area with regards to another site's TOS. But the burden of proof is on them, not the scraper. That and masking your scrapers is too easy.
[2] - Lead generation, selling data, consumer + market research
[3] - I think it is always a gray area with regards to another site's TOS. But the burden of proof is on them, not the scraper. That and masking your scrapers is too easy.
Do you have experience with people using web scraping commercially?
What field or industry was it being used in?
Do you think it fell in a 'grey' area, or was it explicitly disallowed or allowed?
Thanks for any info!