I hate it when I can't get in touch with the right engineers at a large company. This (especially the highly targeted ad mentioned on the page) is a very creative way to try to solve that problem.
Not associated with Meta, but this piqued my interest. That being said, I found some parts confusing and hard to follow. For example what does URPF (Unicast Reverse Path Forwarding) in the title of this submission have to do with the contents?
And is the packet loss supposedly happening at specific times only? It's not mentioned anywhere, but one screenshot highlights the time. I couldn't reproduce the packet loss using any of the looking glasses and dest IP addresses in the screenshots. At this point, if this was a report I had received about one of my services, I would have probably bumped down the priority to low and asked for a reproducible test, because in my experience even issues that affect a single path in an ECMP group are not this hard to reproduce. I think it's way more important to give the engineer who will process the report an easy way to check that there is indeed a problem than to start to teach how traceroute works.
TBF, there does seem to be an issue somewhere, because sticking 129.134.80.234, one of the Meta IP addresses from a screenshot, on ping.pe does definitely show significant packet loss from more locations than you'd expect to see for an address with no connectivity issues.
I'm only familiar with this as a user and not a developer, but I've had multiple Android phone where not all camera features available in the Camera app were available to other apps via the APIs:
BunnyCDN don't run their own network, most of their servers are hosted at DataPacket(.com), but they use some other hosting companies too.
DataPacket has a very large network though and is kind of, sort of EU-based. AFAIK most operations are in Czechia, but the company is registered in UK. And there's also the Luxembourg-based Gcore.
> Worryingly, VNPT and Bunny Communications are home/mobile ISPs
VNPT is a residential / mobile ISP, but they also run datacentres (e.g. [1]) and offer VPS, dedicated server rentals, etc. Most companies would use separate ASes for residential vs hosting use, but I guess they don't, which would make them very attractive to someone deploying crawlers.
And Bunny Communications (AS5065) is a pretty obvious 'residential' VPN / proxy provider trying to trick IP geolocation / reputation providers. Just look at the website [2], it's very low effort. They have a page literally called 'Sample page' up and the 'Blog' is all placeholder text, e.g. 'The Art of Drawing Readers In: Your attractive post title goes here'.
Another hint is that some of their upstreams are server-hosting companies rather than transit providers that a consumer ISP would use [3].
These days RFC8805[0] is pretty widely supported. But as far as I understand, it's not entirely trusted and geolocation providers will still override that data if it doesn't match traceroutes and whatever other sources they use
If you're buying transit, you'll have a hard time getting away with less than 10% commit, i.e. you'll have to pay for 10 Gbps of transit to have a 100 Gbps port, which will typically run into 4 digits USD / month. You'll need a few hundred Gbps of network and scrubbing capacity to handle common DDoS attacks using amplification from script kids with a 10 Gbps uplink server that allow spoofing, and probably on the order of 50+ Tbps to handle Aisuru.
If you're just renting servers instead, you have a few options that are effectively closer to a 1% commit, but better have a plan B for when your upstreams drop you if the incoming attack traffic starts disrupting other customers - see Neoprotect having to shut down their service last month.
From having worked on DDoS mitigation, there's pretty much no difference between CGNAT and IPv6. Block or rate limit an IPv4 address and you might block some legitimate traffic if it's a NAT address. Block a single IPv6 address... And you might discover that the user controls an entire /64 or whatever prefix. So if you're in a situation where you can't filter out attack trafic by stateless signature (which is pretty bad already), you'll probably err on the side of blocking larger prefixes anyway, which potentially affect other users, the same as with CGNAT.
Insofar as it makes a difference for DDoS mitigation, the scarcity of IPv4 is more of a feature than a bug.
This is very challenging, in about one year the biggest recorded DDoS attack has increased from 5 Tbps to almost 30.
Almost all of the DDoS mitigation providers have been struggling for a few weeks because they just don't have enough edge capacity.
And normal hosting companies that are not focused on DDoS mitigation also seem to have had issues, but with less impact to other customers as they'll just blackhole addresses under larger attacks. For example, I've seen all connections to / from some of my services at Hetzner time out way more frequently than usual, and some at OVH too. Then one of my smaller hosting providers got hit with an attack of at least 1 Tbps which saturated a bunch of their transit links.
Cloudflare and maybe a couple of the other enterprise providers (Gcore?) operate at a large enough scale to handle these attacks, but all the smaller ones (who tend to have more affordable rates and more application-specific filters for sensitive applications that can't deal with much leakage) seem to be in quite a bad spot right now. Cloudflare Magic Transit pricing supposedly starts at around $4k / month, and it would really suck if that became the floor for being able to run a non-HTTP service online.
Something like Team Cymru's UTRS service (with Flowspec support) could potentially help to mitigate attacks at the source, but residential ISPs and maybe the T1s would need to join it, and I don't see that happening anytime soon.
Over 22 hours of downtime for the one VPS I have in that region.
My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.
On Firefox on Android on my pretty old phone, a blurry preview rendered in about 10 seconds, and it was fully rendered in 20 something seconds. Smooth panning and zooming the entire time
> One crawler downloaded 73 TB of zipped HTML files in May 2024 [...] This cost us over $5,000 in bandwidth charges
I had to do a double take here. I run (mostly using dedicated servers) infrastructure that handles a few hundred TB of traffic per month, and my traffic costs are on the order of $0.50 to $3 per TB (mostly depending on the geographical location). AWS egress costs are just nuts.
Anecdata, but I've been in Romania for a couple of weeks recently and Uber was cheaper than Bolt for all rides except one. I think I only used Bolt while I had 40% off voucher active, and sometimes even with that it wasn't much cheaper than Uber.
But I believe in the past Bolt used to be cheaper than Uber indeed.
You can change the root parameter in extlinux.conf to point to whatever block storage you want it to
Or reconfigure U-Boot to load extlinux.conf (and the kernel image, initrd, etc) entirely from other type of media. IIRC NVMe, USB and external SDs are all supported
> It's dawning on me that what China and South Korea are achieving, although impressive, leads to a lengthy stalemate that makes life impossible for millions of people.
It's not a stalemate, they're still getting new cases at a pace that allows their medical system to cope.
> In the absence of a vaccine
It's a pretty safe bet one will become available pretty soon. Meanwhile, more is being learned about how to handle infections and improve the outcome.
> the aim is to flatten the curve but still get the whole thing over and done with in about 6 months
There are 4000 ICU beds in the country in total, most of which will be in use due to other kind of cases at any one time. [1] But let's assume you can make that number available for coronavirus patients for 6 months. So you have 4k*26 = 104k ICU bed-weeks available. There's been talk of 60% of the population getting infected to build up herd immunity [2] (somehow ignoring that there seems to be a nontrivial reinfection rate [3]), so almost 40 million people. It's not very clear how many infected people end up needing intensive care. In Italy, it was 10% of the people who tested positive [4]. But only the worst cases get tested once the epidemic is widespread, so let's say maybe 0.5% of the infected people need ICU (wild guess here since no country with a large number of infections is testing people with mild symptoms, but I think I'm being conservative). 0.5% of 40 million is 200k patients. If each of them need an ICU bed for 2 weeks, that's 400k bed-weeks.
Basically we're talking about most of the 6 months period of the NHS being overwhelmed and coronavirus having a high mortality rate.
> In case you mean the .fw file used in that driver, that's a misnomer. There is no CPU inside the EPDC. That .fw file is a plain voltage waveform file that defines voltage vs time waveforms to get pixels to appropriate graylevels. "Liberating" that would be akin to "liberating" the content of a .wav file. ie: just use hexdump -C.
I meant the waveform data at offset 0x700000 on the internal storage of the Kobo devices (which is similar but not the same as the .fw files in firmware/imx/). It's not clear whether distribution is allowed.
FYI, the OEM (ODM?) for Kobo devices is Netronix (http://www.netronixinc.com/index.aspx). On their platforms, the most challenging bits to liberate will be the EPDC firmware (which is specific to each panel model) and E Ink's Regal library (although the latter isn't required - I've dropped it in okreader). I expect that even if you'd get your own devices manufactured, those would still be problematic. You'd probably also want to go for a non-Broadcom WiFi adapter vendor, to avoid runtime firmware loading.
Not associated with Meta, but this piqued my interest. That being said, I found some parts confusing and hard to follow. For example what does URPF (Unicast Reverse Path Forwarding) in the title of this submission have to do with the contents?
And is the packet loss supposedly happening at specific times only? It's not mentioned anywhere, but one screenshot highlights the time. I couldn't reproduce the packet loss using any of the looking glasses and dest IP addresses in the screenshots. At this point, if this was a report I had received about one of my services, I would have probably bumped down the priority to low and asked for a reproducible test, because in my experience even issues that affect a single path in an ECMP group are not this hard to reproduce. I think it's way more important to give the engineer who will process the report an easy way to check that there is indeed a problem than to start to teach how traceroute works.
TBF, there does seem to be an issue somewhere, because sticking 129.134.80.234, one of the Meta IP addresses from a screenshot, on ping.pe does definitely show significant packet loss from more locations than you'd expect to see for an address with no connectivity issues.