The western AI crawlers are generally very-well behaved. GPTBot and ClaudeBot are nice crawlers and I see them respecting my robots.txt.
There are of course plenty of mystery, obfuscated/camouflaged scrapers/crawlers from god knows whom. Thankfully they are easy to spot and ban, although I've definitely thought about deliberating sending them poisoned data.
Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).
When even OpenAI is more respectful of intellectual property and website owner control than you, that's a problme.
Fun and learning is a really good reason. It also reminds me of the damascene: trying to achieve something that doesn't feel possible, and working through all the extreme resource-constrained engineering limits.
I think most of my embedded projects aren't that useful, but they've taught me a lot.
Good. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly.
That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
I don't think there's been any official confirmation, but even Fable safeguards seem to have gotten quite a bit of tuning, is less trigger-happy and less like regex matching.
Are you thinking of single shot? For agentic use cases, your ability to manage cache and context becomes much more important. Price per individual request becomes less about tokens and more about cache.
They charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss).
Anthropic (and now OpenAI too for 5.6) prompt caching is not free.
Unless stated otherwise, opinions here are my own and personal.