Ask HN: How to build services for FOSS projects when ToS forbids scraping?

3 pointsby ATechGuyقبل 11 شهرًا6 comments

I'm working on a service for FOSS developers to help enforce code license compliance and make projects more sustainable.

The challenge: many websites' Terms of Service explicitly prohibit scraping, crawling, or automation. At the same time, the information needed (repos, dependencies, metadata) is often available only through those sites.

For those who've built tools around open source ecosystems:

* How do you navigate ToS restrictions while still delivering value to users?

* Do you focus on official APIs only, even if they're limited?

* Are there established legal/technical best practices for this situation?

* How to balance compliance with ToS and the mission of supporting FOSS?

Curious to hear what others have done (or seen work) in this space.

2 comments

like_any_otherقبل 11 شهرًا

The law around scraping is unfortunately complex, but it's plausible your use of it (legal compliance) could be legal regardless of ToS ("you're not allowed to use automated tools to check if we're breaking your license" is quite fishy):

https://legalclarity.org/is-web-scraping-legal-a-look-at-the...

https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Or you could base your business in Denmark: https://en.wikipedia.org/wiki/Web_scraping#European_Union

ATechGuyقبل 11 شهرًا

Thanks for sharing the links! You're right, the legality around scraping is nuanced :( Use cases like license compliance might fall on the defensible side.

toomuchtodoقبل 11 شهرًا

I’m unsure what your jurisdiction is, but if you’re in the US, speak with an attorney and get an attorney opinion letter. You’ll at least be able to demonstrate that you were acting in good faith.

ATechGuyقبل 11 شهرًا

Thanks!

bigyabaiقبل 11 شهرًا

You're making a business, right? Pay for official API access.

ATechGuyقبل 11 شهرًا

Right now the cost is a blocker (I'm bootstrapped, no funding yet). On top of that, sometimes official APIs either don't exist or don't expose the full dataset needed for license checks, which makes things tricky.