A lot of the discussion around the /crawl endpoint seems to miss a key detail in the docs. The crawler explicitly identifies itself as a bot, respects robots.txt, and does not bypass CAPTCHAs, WAF rules, or Cloudflare Bot Management.
So technically it’s a nice managed crawling system, but in practice it only works on sites that already allow bots to crawl them. For many real-world data extraction use cases, the problem isn’t crawling infrastructure, it’s dealing with sites that actively block bots. In those cases you still need traditional scraping approaches.
Interesting take on it. Some people probably wouldn't like to be called soft but there is likely some truth to it.
I feel it really comes down to priorities.
Scraping has always been a means to a end for most companies. Get data and then use it for something valuable. Before getting the data was easy, but now it is getting increasingly harder.
I think the key here is highlighting the fact that the time of cheap/easy/low skilled access to web data is ending. Companies either need to skill up on understanding how to bypass anti-bots or pay someone else to do it for them and they focus on the data.
One of the main ideas, we explored here is how scraping has shifted from being mainly a technical challenge to an economic one:
- Infrastructure and proxies have gotten cheaper, but anti-bot defenses have evolved fast.
- Because of that, the real cost of scraping is now the cost per successful result, and spikes of 5x–20x can happen when defenses tighten.
- The bottleneck today isn’t just “can you scrape it?”, it’s whether you can do it profitably and efficiently.
I’d love to hear how folks here are dealing with rising scraping costs or what strategies have worked when data value doesn’t obviously outweigh defense costs.
We just dropped the State of Web Scraping 2025 report. TL;DR: scraping is scaling—fast.
- Market boom: Web scraping is growing 15% YoY and projected to hit $13B by 2033. Web data is now a real asset class. The gold rush is on.
- AI + scraping: LLMs are getting surprisingly good at generating spiders, debugging selectors, and auto-healing. Still brittle, but improving fast. 2025 might be the year of the “self-healing” scraper.
- Bot wars intensify: Anti-bot tech is getting aggressive, Cloudflare, decoy pages, forced JS rendering, login walls. Scraping popular sites is now high-effort and high-cost.
- Proxy market shake-up: Residential/mobile proxy prices have dropped 25–50% in the last year, thanks to scrappy newcomers. But domain-level pricing is rising, creating more complexity and less transparency.
- Legal landscape: Lines are getting clearer: public data is generally safe; behind logins is risky. AI crawlers are under increasing scrutiny, and enforcement is likely to tighten.
- Scraping stack evolution: New tools are focusing on stealth, AI assistance, and integration into real data pipelines. The modern scraping stack looks more like infrastructure than hacked-together scripts.
Big picture: 2025 is shaping up to be a turning point. Smarter scrapers, tougher competition, higher stakes.
The ethics of these free VPNs and hidden proxy SDKs are very questionable. But they are crazy profitable for the proxy providers running them so unlikely to go away.
It is this type of attitude that is why websites are becoming so aggressive in blocking web scrapers.
Being an "ethical web scraper" is about your own ethics, not abusing other peoples servers/data, and preserving the open internet for everyone.
Yes, if you slam someones website you will probably get the data you want but it will damage it for the collective. If everyone just slammed websites then:
1) the website owners will just get pissed off and either shut data behind logins or make it inaccessible. This is basically what happened with LinkedIn.
2) make it much harder and costlier to scrape by using more advanced anti-bots, requiring everyone to use more expensive residential proxies and headless browsers to get past them.
Web scraping can be a burden for websites, so everyone should approach it in as responsible and ethical way as possible.
Good point, wouldn't say archiving is unethical at all...I was thinking more along the lines of someone scraping a entire segment of a websites data and reproducing it 1 for 1 on their own site with zero value add.
I think we can't make broad statements saying that web scraping is ethical or unethical, it isn't that black or white. It really depends on what is being scraped, how is it being used, and the intention of the scraper.
Some web scraping can be unethical, say for example if you are scraping a site solely to mirror their content and add zero value to the original content owner.
However, there are a lot of web scraping use cases which are beneficial to the site being scraped and actually add value. Two examples:
- Google: Ahrefs & SEMRush scrape Google so they can provide SEO analytics to companies looking to grow their companies. Googles keyword analytics aren't great, so Google has effectively outsourced providing a good analytics tool to Ahrefs & SEMRush who products increase the value of the Google SERPs ecosystem.
- Amazon + Other E-Commerce: Amazon wants brands and 3rd party stores to list products on their site, and the companies scraping Amazon to provide product placement tools to their users make it easier and more profitable to list products on Amazon. Leading to more and more companies listing products on Amazon.
Interesting!...I'm not a lawyer, so the content for this piece was based on commentary in the below article. Was written by their lawyer, but would love to hear your counter point to it. Always good to get multiple viewpoints on something.
You can do it as a service, but that is highly competitive and basically trading time for money. Best ways are to productize it:
- build a on-demand data api for a specific type of data and charge a premium for it. Good example is https://serpapi.com/ who do Google data, charge ~10X markup on proxy costs
- proxy solutions make good money. To scrape at scale you need proxies, and lots of users pay $1-5k per month. Lots of proxy solutions doing +$100k per month.
- build a tool that uses web scraped data, analyses/filters it and displays it to users. Lots of the biggest web scrapers are doing this, ex. doing product monitoring products for e-commerce companies, etc. Lots of competition there, but you can do it in new markets, like NFTs, etc.
- hedge funds will pay huge money for web data, if you have 5 years of continuous data so they can backtest it.
100% agree, when scraping it should always be done respectfully.
- If they provide a API, then use it.
- Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time).
- If you can get cached data from somewhere that works, then use that.
Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need.
The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
great point - personally, I see so many people wasting massive amounts of time and money on content that goes nowhere. Content producers should approach content like investors and go after the opportunities with the best ROIs
Sharing is caring really. I've analysed numerous content niches with this technique (lots that I haven't produced content for) and there is so much opportunity there that I feel people should take advantage of it.
I haven't checked all the tools, but even out of the paid ones SEMRush is the only one I've found that allows you to export all the keywords in a way to make this technique effective.
So technically it’s a nice managed crawling system, but in practice it only works on sites that already allow bots to crawl them. For many real-world data extraction use cases, the problem isn’t crawling infrastructure, it’s dealing with sites that actively block bots. In those cases you still need traditional scraping approaches.