> But here we're talking about a system where legitimate users (human browsers) and scrapers get the same value for every application of the work function. The cost:value ratio is unchanged; it's just that everything is more expensive for everybody. You're getting the worst of both worlds: user-visible costs and a system that favors large centralized well-capitalized clients.
Based on my own experience fighting these AI scrappers, I feel that the way they are actually implemented makes it that in practice there is asymmetry in the work scrappers have to do vs humans.
The pattern these scrappers follow is that they are highly distributed. I’ll see a given {ip, UA} pair make a request to /foo immediately followed by _hundreds_ of requests from completely different {ip, UA} pairs to all the links from that page (ie: /foo/a, /foo/b, /foo/c, etc..).
This is a big part of what makes these AI crawlers such a challenge for us admins. There isn’t a whole lot we can do to apply regular rate limiting techniques: the IPs are always changing and are no longer limited to corporate ASN (I’m now seeing IPs belonging to consumer ISPs and even cell phone companies), and the User Agents all look genuine. But when looking through the logs you can see the pattern that all these unrelated requests are actually working together to perform a BFS traversal of your site.
Given this pattern, I believe that’s what makes the Anubis approach actually work well in practice. For a given user, they will encounter the challenge once when accessing the site the first time, then they’ll be able to navigate through it without incurring any cost. While the AI scrappers would need to solve the challenge for every single one of their “nodes” (or whatever it is they would call their {ip, UA} pairs). From a site reliability perspective, I don’t even care if the crawlers manage to solve the challenge or not. That it manages to slow them down enough to rate limit them as a network is enough.
To be clear: I don’t disagree with you that the cost incurred by regular human users is still high. But I don’t think it’s fair to say that this is not a situation in which the cost to the adversary is not asymmetrical. It wouldn’t be if the AI crawlers hadn’t converged towards an implementation that behaves as a DDOS botnet.
I find it really frustrating that I am not able to avoid using whatsapp due to how popular it is to the point that it’s become the go-to communication channel for most things :/
Just wanted to call out that linkhut does have IFTTT integration [0]. Although you will need to pay for an IFTTT developer account if you want to enable that on your self hosted instance. If you need help setting that up let me know, I’ll be happy to walk you through the process (and that way I can write documentation on how to do it).
Edit: I also was under the impression that one could use the webhooks integration [1] to make bespoke integrations (as the documentation says they support passing arbitrary request headers) but haven’t tried it myself, have you tried it and run into any issues? I’d also be happy to help improving support for that workflow.
Do you have pointers to this? As a Spaniard I recall that even though I was originally told the spanish alphabet treated the LL as its own letter, it always felt quite inconsistent. And I always assumed it’s removal was more about simplifying things than having to do with computers
I’ve been using a Steel HR for a few years now. I’m extremely satisfied.
Long battery life is an understatement! Even when the battery drops to near 0 the watch is still useable as a plain analog watch for at least a week giving me enough time to find where I left the damn charger (given that I don’t get to use it often at all).
What I don’t get is how this model (analog watch with just enough smarts) hasn’t gained more traction. It seems to me there’s a lot of opportunities in this space, not everything requires a high resolution display.
You might be right that it won’t matter in the long run. But I’d argue that targeting gamers and tech enthusiasts is going to make it harder to reach mass market adoption.
For instance, with computers, laptops and cell phones being first and foremost business tools it was common in the early days for those to be provided by the employers. That really played a big role in normalizing their presence around the home before they became mainstream products.
I have a hard time seeing a similar thing happening with these goggles, but I’m far from being an expert in these things.
> If we go further back in time, mobile phones were bricks used only by business people [...]
But they were widely used by business people, and I’m assuming that was an important aspect of permeating into the other demographics.
To me, what happened with cellphones is similar to what happened with personal computers. It first entered households of people that needed them for work, and as the tech evolved, so did their recreational capabilities making them more appealing to the mass market.
AR/VR devices are attempting this the other way around: their current use cases seem to be mostly recreational, and we are hoping they will eventually come up with some productivity use cases. Someone in this thread talked about how they are surprised that tablets have sort of flopped, and I feel it’s because of similar reasons. Sure there are now more and more ways to be productive on a tablet, but it’s still more of a consume-centric experience and I believe that’s the reason they haven’t replaced laptops. Time will tell.
> could anyone have imagined walking into a living room 30 years ago and seeing everyone hunched over a phone?
True, but I don't recall smartphones ever being unveiled with promotional videos of people all being hunched over them at dinner.
Smartphones were promoted by enabling us to do things on the go: making and receiving phone calls, getting directions, finding restaurants in the area. This is why this feels different: the experience that is promoted is one that feels already slightly dystopian.
Not parent and not arguing that this model is common by any means. But a noteable example is Motion Twin [0]: a game company that had a couple of mainstream hits (most recently Dead Cells).
I’m working on an open-source social bookmarking site in Elixir that is API compatible with delicious/pinboard.
It’s named linkhut and it’s currently able to import your bookmarks from pinboard and browser exports. The flagship instance is: https://ln.ht
I’m still working on the archiving and full text search feature. I’ve been experimenting with different approaches and there’s still a few things I want to explore before settling on a solution.
I get that this is not really useful to you as of now, but if you still haven’t found anything in a couple of months, think about checking it out I might just have launched that feature by then.
The flourishes in the D of that cover are exactly how I was taught to write the uppercase D in cursive at school. I remember (as a kid) thinking it was odd that to write a `D` one had to start by writing an `I`, but never questioned it.
But from looking at examples of cursive on google images, it seems like that form is no longer as prominent.
But since it looks like the proxy fleet is made of a large set of nodes and each one of them only keeps track of their own state, if sourcehut were to only keep one counter for the whole user-agent it would lead to a large number of nodes not being able to refresh their cache for potentially long periods of time (there's nothing that would guarantee for example that it's always a new node that gets to make the one request per hour).
That's precisely why it's so unfair to ask the sourcehut team to try to come up with a solution to this problem: it's the very design of the proxy that Google put together that is causing this issue in the first place. And the sourcehut team has no control over their design. And to add insult to injury: when the sourcehut team offers recommendations on how to improve their implementation, Google responds with "it's too much work".
Still, it makes for a great example of the difference between GPT-3 and an AGI. We would expect the latter to have enough self-awareness to recognize when it is being asked to do something beyond its abilities.
> During development, I noticed that the unicode_string library raises an exception when trying to get the next word if there is invalid UTF-8 anywhere in the string. That seemed weird to me, because I would expect the library to only scan the part of the string that is relevant to get the next word. If the library is checking the whole string for UTF-8 validity every time you want to get the length of the next word, that will lead to non-linear algorithmic complexity.
I wonder: could the algorithm implemented in unicode_string be improved? Or were the algorithms implemented as Rust NIFs also prone to this non-linear complexity?
Based on my own experience fighting these AI scrappers, I feel that the way they are actually implemented makes it that in practice there is asymmetry in the work scrappers have to do vs humans.
The pattern these scrappers follow is that they are highly distributed. I’ll see a given {ip, UA} pair make a request to /foo immediately followed by _hundreds_ of requests from completely different {ip, UA} pairs to all the links from that page (ie: /foo/a, /foo/b, /foo/c, etc..).
This is a big part of what makes these AI crawlers such a challenge for us admins. There isn’t a whole lot we can do to apply regular rate limiting techniques: the IPs are always changing and are no longer limited to corporate ASN (I’m now seeing IPs belonging to consumer ISPs and even cell phone companies), and the User Agents all look genuine. But when looking through the logs you can see the pattern that all these unrelated requests are actually working together to perform a BFS traversal of your site.
Given this pattern, I believe that’s what makes the Anubis approach actually work well in practice. For a given user, they will encounter the challenge once when accessing the site the first time, then they’ll be able to navigate through it without incurring any cost. While the AI scrappers would need to solve the challenge for every single one of their “nodes” (or whatever it is they would call their {ip, UA} pairs). From a site reliability perspective, I don’t even care if the crawlers manage to solve the challenge or not. That it manages to slow them down enough to rate limit them as a network is enough.
To be clear: I don’t disagree with you that the cost incurred by regular human users is still high. But I don’t think it’s fair to say that this is not a situation in which the cost to the adversary is not asymmetrical. It wouldn’t be if the AI crawlers hadn’t converged towards an implementation that behaves as a DDOS botnet.