Prosopo could cut out a lot of the residential proxy nonsense for you. We integrate with lambda@Edge / cloudfront workers / server side and perform analytics to detect residential proxy networks - at far less cost than DD or HUMAN.
If you're on a proxy network then you're detectable by TLS handshake timings, for example. You are right though that agent mouse and touch behaviour is almost inseperable from human data these days. So you need to use a combination of techniques in order to detect automation. At Prosopo we check TLS timings, SIMD performance, page similarity, fingerprint proofs, agent honey pots, JS inconsistencies. Rate limiting is surprisingly effective as an initial deterrant. There is no magic single thing that works for all clients, it's a case of learning distinct bot operators' behaviours.
> The site’s content happens to include an HTTP/2
endpoint that, when presented with an anonymous authorization token, behaves as a VPN server’s outer layer
Is this not the fingerprintable aspect? Wouldn't you need to randomise the HTTP endpoint to avoid eventually being banned based on URI?
Captcha are already expensive at scale due to escalating checks when abuse is detected. You have to orchestrate and pay for residential proxies, containers with different fingerprints, different behavioural data, clean IP rep, emulate device performance to avoid revealing youre running on a server... A 1-shot doesn't scale against this.
This is the standard now for astroturfing online. Build up a profile over time with varied interactions, sometimes over years, and then sell it for a few hundred dollars via blackhatworld. I've not seen hn listed but reddit definitely follows this pattern.
If you think the IPs are normal, you can check if people are proxying by looking at DNS connecting IP (they may not have proxied UDP), SIMD score (server CPUs cluster differently to consumer), residential proxy lists (there are a bunch of these), invalid webgpu setups, etc. Maybe this kind of detection is against HN way of doing things but I've definitely seen recaptcha on the login before and it employs a bunch of these checks. Happy to help!
So far its cost me $2.27 to submit a contact form 3 times - why is this better than a captcha solver with human solves at 1000 per $2?
On your automation, your tool fed back to me as follows after 3 submissions:
> The CAPTCHA is persistently blocking now — Prosopo's widget appears to have flagged the session/IP due to the repeated submissions. The checkbox won't reset this time. This is expected behavior from their bot protection product. To submit again, you'd likely need to wait a while for the rate limit to clear, or submit manually from your own browser.
It's not hard to setup JA4 monitoring and I think its valid as a coarse filter. There are various plugins for nginx/node.
> I've seen people waste so much time with the whack a mole JA4 block just because they like the intellectual challenge
You just store the ja4 on requests and build a catalogue of known JA4s over time using statistics. Outlier JA4s you treat with suspicion by default and challenge. It shouldn't be manual.
> If someone invests time/money in using a captcha solver, they're already dedicated enough and will easily get around a JA4 signature block.
Obviously, not for the regular user but captcha solvers are also blockable:
- proxy detection
- detection by running DNS server and capturing real IP over UDP request
- abnormal TLS handshake latency
- repeat behaviour at scale
- rendering captcha on a fake origin instead of in the real page
In combination with other signals JA4s are useful. You learn to spot obviously incorrect ones because Chrome always looks different from Safari which looks different to Firefox. Captcha solvers have their own unique JA4s based on whatever scripting language they're using (pyhton / rust / node). As another commentor pointed out, browsers have unique sets of headers like priority, DNT. So yes, it won't stop dedicated attackers but it is worth implementing as a coarse filter.
At the time, reCAPTCHA was the alternative and it was effectively working as a giant ad targeting data collection tool. I'm pretty sure Google have now back tracked from this.
WebGL finger printing is just one of many things you need to do if you actually want to stop automation. There is no way round it other than requiring ID of some sort.
I think we're talking about 2 different things. PoW is annoying for basic scrapers but it really doesn't affect enterprise grade bot operations with access to unlimited residential proxies.
It's either that or you tie tickets to government ID like in France. If the arbitrage opportunity is more than the cost of automation then someone will exploit it.