I use my home connection for VPN access remotely. I back up snapshots of data every day. I like to be able to download games and Linux ISOs practically on demand. I work from home and often enough faster speeds can avoid several minutes of additional waiting in a day.
This connection is shared as well. My partner relies heavily on cloud syncing. We both like to stream 4K HDR video. I like being able to get devices updated and ready to use with minimal time spent waiting for downloads.
I also live in NZ, where multi-gigabit fibre connections are often cheaper than what Americans have to pay for a fraction of the bandwidth. It’s not a notable financial burden or anything, and it’s not like we have data caps to worry about. It’s very much a situation where the use cases naturally find themselves once the option is there.
Also, 25/10 Mbps is painfully slow for a shared home connection in the modern day. There’s videos on YouTube that can push a higher bitrate than that. The absolute slowest plan that my ISP even offers is 100/20 Mbps for about $35 USD per month, while the most common/baseline plan for most households in NZ is 500/100 Mbps after the fibre carriers continued to increase speeds at the lowest tiers.
I love seeing trees in more places, but for parking lots in particular they do have some downsides compared to solar panels. They often require more space; they attract birds that that poo on vehicles; and there’s a higher risk of collateral damage during windstorms. Not to mention that solar panels directly produce electricity, of course.
We absolutely should see more trees in many cities, but they introduce their own challenges in parking lots, especially if they’re placed retroactively.
The extent of that heavily depends on where you are. Where I live in NZ, the grid export rates are very low while the import rates are very high.
Our peak import rate is 3x higher than our solar export rate. In other words, we’d need to sell 3 kWh hours of energy to offset the cost of using 1 kWh at peak.
We’re currently in the process of accepting a quote for home batteries. The rates here highly incentivise maximising self-use.
Automatic updates are absolutely not peak stupidity. Most users’ devices would have nasty security vulnerabilities wide open for a much longer period of time without automatic updates.
Hey. I run a small community forum and I've been dealing with this exact same kind of behaviour where well over 99% of requests are bad crawlers. There used to be plenty of "tells" for the faked browsers, HTTP/1.1 being a huge one. As you said, however, they're getting a bit smarter about that and it's becoming increasingly difficult to differentiate it from legitimate traffic.
It's been getting worse over the past year, with the past few weeks in particular seeing a massive change literally overnight. I had to aggressively tune my WAF rules to even remotely get things under control. With Cloudflare I'm aggressively issuing browser challenges to any browser that looks remotely suspicious, and the pass rate is currently below 0.5%. For my users' sake, a successful browser challenge is "valid" for over a month, but this still feels like another thing that'll eventually be bypassed.
I'd be keen to know if you've found any other effective ways of mitigating these most recent aggressive scraping requests. Even a simple "yes" or "no" would be appreciated; I think it's fair to be apprehensive about sharing some specific details publicly since even a lot of folks here on HN seem to think it's their right to scrape content with orders of magnitude higher throughput than all users combined.
I really don't know how this is sustainable long-term. It's eaten up quite a lot of my personal time and effort just for the sake of a hobby that I otherwise greatly enjoy.
Thank you for speaking some sense. As a site operator that's been inundated with junk traffic over the past ~month where well in excess of 99% of it has to be blocked, the scrapers have brought this upon themselves.
I actually do let quite a few known, "good" scrapers scrape my stuff. They identify themselves, they make it clear what they do, and they respect conventions like robots.txt.
These residential proxies have been abused by scrapers that use random legit-looking user agents and absolutely hammer websites. What is it with these scrapers just not understanding consent? It's gross.
I run a really small forum and I've been absolutely inundated with a bunch of junk traffic. I had to tighten my Cloudflare WAF rules a whole bunch, and start issuing browser challenges way more aggressively.
Excluding known "good" crawlers, well over 99% of the traffic trying to hit the site has been attempting to maliciously scrape. Most of this traffic looks genuine, but has random genuine-looking user agents and comes from random residential proxies in various countries, usually the US.
For the traffic that does make it all the way to a browser challenge, the success rate is a measly 0.48%. Put another way, over 50% of traffic is already blocked by that point, and of the under 50% that makes it to a browser challenge, more than 99.5% fails that challenge.
It's been virtually no disruption to users either, since I configured successful challenges to be remembered for a long period of time. The legitimate traffic is a gentle trickle, while the WAF is holding back garbage traffic that's orders of magnitude above and beyond normal levels. The scale of it is truly insane.
For many ephemeral workloads, sure, but that comes at the expense of generally worse and less consistent CPU performance.
There are plenty of workloads where I’d love to double the memory and halve the cores compared to what the memory-optimised R instances offer, or where I could further double the cores and halve the RAM from what the compute-optimised C instances can do.
“Serverless” options can provide that to an extent, but it’s no free lunch, especially in situations where performance is a large consideration. I’ve found some use cases where it was better to avoid AWS entirely and opt for dedicated options elsewhere. AWS is remarkably uncompetitive in some use cases.
I like that https://discordstatus.com/ shows the API response times as well. There's times where Discord will seem to have issues, and those correlate very well with increased API response times usually.
Reddit Status used to show API response times way back in the day as well when I used to use the site, but they've really watered it down since then. Everything that goes there needs to be manually put in now AFAIK. Not to mention that one of the few sections is for "ads.reddit.com", classic.
I set up Immich last week and I absolutely love it. Docker is my "happy place" and I found the setup pretty straightforward, though it does have some rough edges that I anticipate will be sorted out as the project continues to mature.
I showed Immich to my partner and they loved it so much that we've ordered significantly more storage for the server to accommodate it. We're currently using both Google Photos and OneDrive, but with this we'll be ditching OneDrive and filling that niche with Immich (as well as expanded network storage in general).
The website and documentation is super clear about not using it as the only source of photos. This is why we'll keep using Google Photos, and why I'll also be backing up Immich and portions of the network storage to B2 via restic. I've used this snapshotting pattern for my general server data for years, and it's even saved me a couple of times. Backups are something you hope to never need to use, but boy are they satisfying when you do need to use them and have them set up properly!
This is similar to my setup and experience as well. I have a multi-purpose server running Samba with 10 Gbps SFP+. I have 2.5 GbE on my desktop and easily saturate that when transferring files.
I even use the network for stuff like old game ROMs and ISOs for emulation. The random access times are still magnitudes better than I’d be able to get with any HDD.
I set up a self-hosted FreshRSS instance as well, and I feel the same benefits from it. I got back into RSS after Reddit pushed its API changes through, and it’s been so refreshing. I’m using social media a lot less now and those tingly FOMO feelings all but disappeared.
You can engine brake with a lot of automatic vehicles as well, since they’ll still have semi-automatic modes. I drive a car with a CVT and I’m able to “down-shift” for engine braking by using fake gears at specific ratios.
This connection is shared as well. My partner relies heavily on cloud syncing. We both like to stream 4K HDR video. I like being able to get devices updated and ready to use with minimal time spent waiting for downloads.
I also live in NZ, where multi-gigabit fibre connections are often cheaper than what Americans have to pay for a fraction of the bandwidth. It’s not a notable financial burden or anything, and it’s not like we have data caps to worry about. It’s very much a situation where the use cases naturally find themselves once the option is there.
Also, 25/10 Mbps is painfully slow for a shared home connection in the modern day. There’s videos on YouTube that can push a higher bitrate than that. The absolute slowest plan that my ISP even offers is 100/20 Mbps for about $35 USD per month, while the most common/baseline plan for most households in NZ is 500/100 Mbps after the fibre carriers continued to increase speeds at the lowest tiers.