Disclosure: I am part of the mostly volunteer run OpenStreetMap ops team.
Technically we able to block and restrict the scrapers after the initial request from an IP. We've seen 400,000 IPs in the last 24 hours. Each IP only does a few requests. Most are not very good at faking browsers, but they are getting better. (HTTP/1.1 vs HTTP/2, obviously faked headers etc)
The problem has been going on for over a year now. It isn't going away. We need journalists and others to help us push back.
We don't offer 2x raster tiles because we simply don't have the resources to do it while keeping the tiles minutely updated and open access. We serve around 60k req/sec on a tiny donation backed budget.
The raster tiles are primarily for our OSM mappers to see their map changes rendered quickly to improve the feedback loop.
OP here. The toot was my sarcastic response after having to rate-limit and block another set of abusive scrapers aggressively hitting our website and mapping API. robots.txt be damned.
OpenStreetMap data is free to download. We published minutely on https://planet.openstreetmap.org/ and the data available via AWS S3 + torrent.
Our data is already publish via torrent, see https://planet.openstreetmap.org
Our data, including minutely updates is available via public S3 buckets (EU & US) supported by the AWS Open Data Sponsorship Program.
We publish a live stream of minutely updated OpenStreetMap data in ready do digest form on https://planet.openstreetmap.org/ and S3. Scraping of our data still happens.
Our S3 bucket is thankfully supported by the AWS Open Data Sponsorship Program.
FYI: https://github.com/mchehab/rasdaemon (replaced mcelog) is the daemon for watching for these ECC and other kernel reported "Reliability, Availability and Serviceability" errors.
rasdaemon also attempts to report which physical DIMM / slot triggered the ECC error.
Sub-domain takeover attack. The sub-domain was CNAME'ed to a S3 bucket and the S3 bucket had likely been deleted. The porn purveyor, re-created a new S3 bucket with pr0n.
The vandalism was reverted in 2 hours over 20 days ago. If Mapbox had stayed current with the diffs we publish, the vandalism would have been gone in a blink.
Correct. The vandalism was reverted within 2 hours and a data update publish a minute or 2 later. Mapbox import the original vandalism change only today for some reason and there seems to have been a delay in importing the revert.
Pretoria was mostly mapped by a single mapper very early in the project. There wasn't high quality aerial imagery available when he literally cycled every street in Pretoria.
Because the map "looks complete" Pretoria hasn't really attracted many mappers since his departure.
Cape Town has a much more vibrant community of mappers. I've also been able to get crazy high resolution aerial imagery for CT from the municipality which helps: http://coct.aerial.openstreetmap.org.za/
Technically we able to block and restrict the scrapers after the initial request from an IP. We've seen 400,000 IPs in the last 24 hours. Each IP only does a few requests. Most are not very good at faking browsers, but they are getting better. (HTTP/1.1 vs HTTP/2, obviously faked headers etc)
The problem has been going on for over a year now. It isn't going away. We need journalists and others to help us push back.