The Chrome extension approach may shift some (most?) of the risk to the end user, since technically they are now the one scraping. Theoretically getdex would be relatively better off in this arrangement, while putting their customers into a legal gray area.
Currently, yes. However it's quite early and I'm exploring whether this is the best option.
The reason I was posting here on HN is to see if anyone more technical may be interested in this as well. E.g. to leverage an API like this with AI agents or otherwise.
When building a SaaS with a Go backend, it's nice to be able to have the option of the agents and workflows being in the same process. And being confident in the ability of that to scale well.
While it's true that Go lacks good ML libraries, for some this isn't too consequential if your app is primarily using Anthropic or OpenAI and a database that offers semantic or hybrid search for RAG. The ML is done elsewhere. Plus it could be that you can leverage MCP servers and at that point you're language agnostic.
Regarding the concurrency model approach with Go and agents, I initially baked a message based approach (a la the Actor model, with one goroutine per agent) into Dive Agents, but eventually found that this would be better implemented as another layer. So currently in Dive it's the user's choice on how to implement concurrency and whether to use messaging. But I anticipate building that back in as an optional layer.
Risor is an embedded scripting library and CLI for Go. This release adds new modules for web crawling, background scheduling, and more. Happy scripting.
Just a quick release announcement for Risor, an embedded scripting library and CLI for Go. Plenty of additions relating to web crawling, background scheduling, and more. Happy scripting.
I'm definitely with you that sites should be leveraging CDNs and similar. But I get that many don't want to do any work to support bots that they don't want to exist in the first place.
To me it seems like the companies actually doing the crawling have an incentive to leverage centralized caching. It makes their own crawling faster (since hitting the cache is much faster than using Playwright etc to load the page) and it reduces the impact on all these sites. Which would then also decrease the impact of this whole bot situation overall.
An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week.
This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actual site.
Even if each company that operates a crawler cached pages across multiple runs, I'd expect a large improvement in the situation.
For more dynamic pages, this obviously doesn't help. But a lot of the web's content is more static and is being crawled thousands of times.
I built something for my own company that crawls using Playwright and caches in S3/Postgres with a TTL for this purpose.
Does this make sense to anyone else? I'm not sure if I'm missing something that makes this harder than it seems on the surface. (Actual question!)
Nice. I was wondering if r53 latency-based routing would work as an optional add-on to this. I didn't look into it but was thinking it might not work because the latency based routing records are A records that have to point at a fixed IP address right (rather than the function url in my case). But presumably API gateway just gives you hostnames as well so maybe I'm mistaken here?
Aha, indeed. A few interesting notes on this. And thanks for the link to your Makefile.
I see you're a Cloudformation Stacks user (good stuff), so you may know some of what I say below already, but maybe the Terraform specific notes will be of interest.
Many AWS deployments (Terraform or otherwise) are single region unfortunately, with a primary factor there being that AWS APIs are for the most part region-specific. The AWS Terraform provider was built with this mindset too, unfortunately. Then the situation is complicated by Terraform's lack of support for dynamic providers, which is one of the most discussed still-open issues in Terraform.
For Burrow here, I was was focused on shipping something and didn't want to spend time on generating that code just yet. But the project has gotten some attention so if this is a recurring request I'll probably do it.
Note that I intentionally chose the 17 AWS regions that are default enabled in every new AWS account, so it should only be if you want to customize it that this becomes important.
A Cloudformation Stacks based deployment to achieve multi-region would be another good (and perhaps better) option here. I could add that to the Makefile, with inspiration from your link there.
Cloudflare Workers are great in general. The way they route client requests to the nearest Cloudflare data center means that a single client will most often end up with a small number (perhaps even 1) of public IP addresses coming out of Cloudflare. And to my knowledge, there's not a way to ask for your client to be routed in a different way. So due to this, AWS Lambda's inherent region-specific behavior is actually an advantage for Burrow, since we can ask to be routed through a specific region (no matter how distant) by virtue of hitting the per-region Lambda URLs.
That said, you could actually layer the two together if you were so inclined, and/or optionally route through Cloudflare and AWS.
Making the Cloudflare Worker implementation of this would be trivial and there's no dependency on Go in any way for the proxy itself.
You might be interested in pgEdge Cloud's new free offering, which includes 3 vCPU, 12 GB storage, and a 3 region cluster.
This is distributed, multi-region, multi-writer PostgreSQL.
The situations where our approach resonates with people the most:
* Ultra-high availability for SaaS
* Low-latency access, especially in multi-region scenarios
* Meeting data residency requirements
* Use with edge platforms like Cloudflare Workers
Our engineering team is a mix of folks that were previously at AWS RDS, plus others coming from a cloud security and IaC background. We're really interested in getting feedback from the community and would love you to give our free "Developer Edition" of pgEdge Cloud a try.
You can check out my blog post to learn more about the launch and see links to our various open source projects.
Keep in the Go ecosystem, retain compatibility with the Go programs you already have, but have a much more concise scripting capability at your disposal.