A Facebook crawler was making 7M requests per day to my stupid website(coding.napolux.com)
coding.napolux.com
A Facebook crawler was making 7M requests per day to my stupid website
https://coding.napolux.com/a-facebook-crawler-was-making-7m-requests-per-day-to-my-stupid-website/
407 comments
We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks.
I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem.
I found out that there is an option in the FB Catalog manager that lets FB auto-remove items from the catalog when their destination page is gone. Disabling this option solved the issue.
Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834
The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.
The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.
I once made a PHP page which fetches a random wikipedia article, but changes the title to "What the fuck is [TOPIC]?". It was incredibly funny, until I started getting angry emails from people, mostly threatening a form of libel lawsuit.
Turns out, since it was a front for all of wikipedia[1], google was agressively indexing it, but the results rarely made it to the first search page. And since this isn't exactly an important site, old results would stick around.
Hence a pattern:
1. Some Rando creates a page about themselves
2. Wikipedia editors, being holy and good, extinguish that nonsense
1.5. GOOGLE INDEXES IT
3. Rando, by nature of being a rando, googles themselves, doesn't find their wikipedia page anymore (that's gone), but does find a link to my site in the first page of google results with their name.
4. Lawyers get involved, somehow
Details: http://omershapira.com/blog/2013/03/randomfax-net/
[1] Hebrew wikipedia. I can't imagine what would've happened on the English version.
Turns out, since it was a front for all of wikipedia[1], google was agressively indexing it, but the results rarely made it to the first search page. And since this isn't exactly an important site, old results would stick around.
Hence a pattern:
1. Some Rando creates a page about themselves
2. Wikipedia editors, being holy and good, extinguish that nonsense
1.5. GOOGLE INDEXES IT
3. Rando, by nature of being a rando, googles themselves, doesn't find their wikipedia page anymore (that's gone), but does find a link to my site in the first page of google results with their name.
4. Lawyers get involved, somehow
Details: http://omershapira.com/blog/2013/03/randomfax-net/
[1] Hebrew wikipedia. I can't imagine what would've happened on the English version.
Doesn't ignoring headers and taking someones site down fall afoul of the CFAA? Especially given how comments here are showing that this is a recurring issue? Depending on which side of the fence you're on, there could be standing to go after FB for this either for money, or for their lawyers to help set a precedent to limit CFAA further.
That's 81 request per second on average.
Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages?
That sounds like a lot of bandwidth (and server stress).
Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages?
That sounds like a lot of bandwidth (and server stress).
Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c
"face:b00c"
"face:b00c"
Related thread at SO:
https://stackoverflow.com/questions/49577546/facebook-crawle...
Reading through the article and comments here I find it odd that a company like FB can't get a crawler to respect robots.txt or 429 status codes.
Even I would stop and think "maybe I should put in some guards against large amounts of traffic" when writing a crawler, and I'm certainly not one of those brilliant minds who manage to pass their interview process.
Even I would stop and think "maybe I should put in some guards against large amounts of traffic" when writing a crawler, and I'm certainly not one of those brilliant minds who manage to pass their interview process.
It's an interesting thing to think about, that someone can make a request to your website and you pay for it.
This is not how the mail works, where someone needs to buy a stamp to spam you.
This is not how the mail works, where someone needs to buy a stamp to spam you.
This reminds me of a case years ago when I had a small-ish video file on my server that would cause Google Bot to just keep fetching it in a loop. It wasted a huge amount of bandwidth for me since I only noticed it in a report at the end of the month. I had no way of figuring out what was wrong with it, so I just deleted the file and the bot went away.
Was it a real FB crawler or a random buggy homemade bot masquerading as one?
Check here: https://developers.facebook.com/docs/sharing/webmasters/craw...
Check here: https://developers.facebook.com/docs/sharing/webmasters/craw...
If you can verify it's actually coming from Facebook's IP address (as many other comments here suggest), you should absolutely get in touch with them, though I'm not sure how. Perhaps their security team. That's a very serious bug that they'd be glad to catch -- if it's affecting you it's surely affecting others.
Otherwise it's a bot/malware/etc. spoofing Facebook and gone wrong, which sucks. And yeah just block it by UA, and hopefully eventually it goes away.
Otherwise it's a bot/malware/etc. spoofing Facebook and gone wrong, which sucks. And yeah just block it by UA, and hopefully eventually it goes away.
This is one of the better HN threads in a while. Starting from an easily described problem, it goes into a number of interesting directions including how FB seems to DDOS certain sites for reasons unknown, defenses against such attacks and DDOS in general, GPL licensing, how to collapse HN conversations, etc.
Not a profound comment. But it was really fun to follow the path down the various rabbit holes. This is why I like HN.
Not a profound comment. But it was really fun to follow the path down the various rabbit holes. This is why I like HN.
Interesting. According to the Facebook Crawler link provided by the OP [1], it makes a range request of compressed resources. I wonder if the response from the OP’s PHP/SQLite app isn’t triggering a crawler bug.
Maybe Cloudflare can step up and act as an intermediary to isolate the problem. Isolating DoS attacks is one of their comparative advantages.
[1] https://developers.facebook.com/docs/sharing/webmasters/craw...
Maybe Cloudflare can step up and act as an intermediary to isolate the problem. Isolating DoS attacks is one of their comparative advantages.
[1] https://developers.facebook.com/docs/sharing/webmasters/craw...
It would be nice to understand how you came to the conclusion that this was a Facebook bot.
Make sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.
Did you check the ip and not just the UA? Does it matches with the one from fb?
Surprise, Facebook not caring about playing nice with others!
Anyway, what reason would they have for so many requests? Why would they need to do 300 per second?
Anyway, what reason would they have for so many requests? Why would they need to do 300 per second?
We had that with google in the past. We put a kml database on a web server and made the google maps API read them. Google went to get them without any throttling (my guess is that they primed their edge servers straight from us), the server went down so fast we sought there had been a hardware failure. We ended up putting a cdn just for google servers.
What would happen if you created a "rabbithole" for webcrawlers? Would they fall for it?
Maybe someone at Facebook is testing a project that didn't work as expected?
Are you sure it's a crawler and not a proxy of some sort? Eg. one of your links is on something high traffic in facebook, and all requests are human, running through fb machines.
Facebook caches URLs pretty aggressively (often you need to explicitly purge the cache to pick up updates) so I don't quite understand exactly what happened here. The article is very light on details. Do you have 7 million unique URLs that have been shared on Facebook? Or is it the same URL being scraped over and over?
Are you correctly sending caching headers?
Are you correctly sending caching headers?
When I enabled IPv6 traffic at home and logged default denies I constantly see Facebook owned IPv6 blocks trying to reach addresses browsing Facebook.
I have no idea what its trying to do but its legitimately the only IPv6 inbound traffic that isn't related to normal browsing.
I have no idea what its trying to do but its legitimately the only IPv6 inbound traffic that isn't related to normal browsing.
I’m a bit confused. Facebook is social media company, why do they send out crawlers?
Why would a crawler want to crawl the same page every second (let alone 71 times a second). Even if I were Google, wouldn't crawling any page every few mins (or say every minute) be way more than enough for freshness?
There was a very old post: https://news.ycombinator.com/item?id=7649025, lots of ways to abuse Facebook network.
> I own a little website I use for some SEO experiments [...] can generate thousands of different pages.
Do you have more details on this? One could read that as if the page is designed to attract crawlers.
Do you have more details on this? One could read that as if the page is designed to attract crawlers.
Is it worth duplicating the fb specific parts of your website at another web address as a test site, so that If it was similarly affected, you could provide the address info to help troubleshoot?
Shouldn't have Cloudflare considered this a DDOS attack?