True, most of the blacklists systems today aren’t realtime like Amazon WAF or Cloudflare.
We need a Crawler blacklist that can in realtime stream list deltas to centralized list and local dbs can pull changes.
Verified domains can push suspected bot ips, where this engine would run heuristics to see if there is a patters across data sources and issue a temporary block with exponential TTL.
There are many problems to solve here, but as any OSS it will evolve over time if there is enough interest in it.
Costs of running this system will be huge though and corp sponsors may not work but individual sponsors may be incentivized as it’s helps them reduce bandwidth, compute costs related to bot traffic.
After the first five minutes of using it on Ubuntu, it crashed with error saying I don't have enough free memory, quick look into system stats proved that wasn't the case.
Anyway, not a great first impression. I guess I'll try again in a few months.
Socials:
- github.com/snehesht
---