I'm a lawyer that works in the web-scraping space, and I always chuckle when I read threads like this. Almost every company that we now consider a monopolist (or their affiliates) in the tech space used scraping a part of their process to build their business, and almost every one of those same monopolists now prohibits startups and competitors from scraping their data (which, invariably, is not actually "their" data in any sort of legally cognizable sense). And so perhaps the ethics of web scraping are not so straightforward. And neither are the legal issues associated with it.
I wrote an article about that last fall that got some attention here.
There are probably two dozen specialists on this issue nationwide. A properly targeted Google search will outperform your personal network of attorneys (and their network of attorneys) 99.9% of the time.
Speaking of surgery, here's an analogy that might be helpful.
Five years ago I had a catastrophic ankle injury I suffered while running in Moab. Two broken bones, lots of torn ligaments, and otherwise irreparable damage without serious surgery.
I interviewed a whole bunch of surgeons before I decided where to go under the knife. And I don't remember where I got the advice, but someone told me the most important question to ask is: "How many times have you performed this specific surgery (a Maisonneuve fracture repair)?"
I eventually found the Steadman clinic and a doctor who had already performed the exact surgery I needed nearly 100 times. Everyone else's answer was less than 5. Some even answered 0. The surgical clinic I used had signed pictures of professional athletes all over the wall. I found the true specialist, and I'm very thankful that I did.
Even bad lawyers and surgeons are expensive. When you have a bet-the-business legal issue, do plenty of advanced interviewing to make sure that the one you hire has plenty of experience with the exact issue you need help with. If it's not obvious that you've found the right person, keep looking.
There are certain tasks that you would never do yourself, such as surgery. And there are certain tasks like changing your oil where anyone with patience and diligence can do it themselves. It's just a question of whether you want to spend the time to learn it and do it.
Responding to a C&D letter from a Fortune 50 company, I would posit, is more like performing surgery than changing your oil. The cost of being wrong is rather high.
It is worth pointing out that FB has a long history of litigating this issue, from Power Ventures, to BrandTotal, to Octoparse, to Voyager Labs and more. And they have about 80-90% track record of success. Obviously, they send C&Ds more often than they file litigation. But to act as if this is a completely idle threat is naive.
I can't share client-specific stories because that's protected by AC privilege. But I think the recipients of these letters sometimes have more potential to negotiate than they realize.
The problem with this advice is that this is a very niche area of law. Unless the people you know have had prior experiences with data-access/web-scraping legal issues, a generalist recommendation is very unlikely to be helpful here.
As an attorney who has experience responding to Meta’s “anti-scraping team,” I think there might be more opportunities for amicable resolutions than you might expect (depending on the specifics of what you’re doing, of course). Meta is not oblivious to the fact that they’re under significant social and regulatory scrutiny. They sometimes play nice if you’re willing to accommodate certain considerations.
Either way, my recommendation would be to find an attorney with industry-specific expertise to address the norms of your industry. C&Ds range from idle shake downs to definite pre-cursors to litigation. Without industry-specific knowledge, it’s hard to know which is which.
No, it is not. If you do not have a fiduciary relationship with Boeing and you have no confidentiality obligations with respect to the information, you are not trading on inside information. If you're in the plane when the door blows up, you're just the first person with material public information. You're not trading on material non-public information.
Scraping other sites while prohibiting it on your own is "do what I say, not what I do" behavior, which I think is a fair, consensus understanding of what it means to be hypocritical.
I think the difference is that defeating the other team is the point of sports, whereas at least ostensibly the law is supposed to provide a set of coherent rules for businesses to compete against each other. Trademarks are defined according to certain legal rules, and if you have one, this is how they provide you with a limited monopoly in a certain context. Allowing businesses to define property law through contracts lets people define the rules however they want. And that leads to irrational results.
I wrote an article about that last fall that got some attention here.
https://news.ycombinator.com/item?id=37264676