Do you really need residential ip for this? for my news scraping i have choose static datacenter proxy first, rotating only as fallback, direct as last. flaresolverr covered cloudflare js challenge, but it wedge on memory time to time, so i add healthcheck for it. on my own site aws waf meet datacenter ua with captcha, so i understand other side too. residential never was necessary for me, but maybe my targets are just easier, what do you scrape?
Do you really need to provide AI full access for postgres? I rather created a small set of fixed tools, rather then raw sql. Each tool has bunch of predefined params and limited context. Each has own purpose: query explain and analize, db monitoring, slow log queries, db healthcheck, backups, etc.
I have choose pgbouncer for my small db, because it does one thing and does it good - transaction pooling, other solutions seemed too complicated for me. All that features which should keep you allow to use listen/notify and set was unnecessary for me, i solved it on code level
i see blocked status for my website, which is expected, because aws waf meet any suspicious(datacenter or non regular ua) with a captcha challenge. How to properly add you bot to exceptions? could you provide a list of ip publicly maybe, just an idea
i hit the same 400 querying the algolia, a raw > in the query string is what does it, encode it as %3E and points>100 comes back fine for me right now. curl -G with --data-urlencode handles the escaping if youd rather not do it by hand
never got a name, best we managed was network attribution. ours came mostly from tencent cloud plus a second hetzner asn, all spoofed chrome on windows user agents, and it rotated from digitalocean to tencent within a day once we started blocking asns. need to tell that it executed the AWS WAF js challenge and replayed the token, but never solved an interactive captcha, so flipping the action from challenge to captcha is what finally stopped it
two different things are getting called "cursor" here. queryset.iterator() isn't server-side row-by-row processing, it's one set-based query streamed to the client in chunks so a big export doesn't materialize in memory all at once, "get it in one go" is exactly the OOM it avoids.
the honest knock on the streaming kind isn't "bad design", it's that it holds a portal open server-side, which is why it can't survive transaction pooling. so the pooler-friendly fix isn't one giant fetch, it's keyset pagination (where id > last limit n), stateless and constant memory, which is what we moved those paths to.
we moved our django app behind pgbouncer transaction pooling a few days ago and the surprise wasn't SET so much as queryset.iterator(). it relies on server side cursors, which don't survive being pooled, so we had to disable it everywhere and let it fall back to client side. also had to move statement_timeout out of the app's connection options into the pooler's own connect query, since libpq startup params just get silently ignored behind it.