My mother-in-law fell for a love scam last year. Luckily we caught it early and no money was lost, but I was surprised how difficult it was to convince her it was a scam despite the obvious signs and family and friends all telling her it was. The scammer claimed to be the actor Jack Wagner (popular on Hallmark channel and soap operas). Out of desperation we looked him up on cameo and paid him for a video message telling her it wasn't him texting her, and that finally snapped her out of it.
Ah, good to know, I hadn't seen that V2 update. Looks like a min 5m inactivity to auto-pause (i.e., scale to 0), and any connection attempt (valid or not) resumes the DB.
Yes, we've been testing volume attachments every 5m since start of the year, and have experienced 100-150 attachment failures per volume type in that time frame during multiple events (most recently last week):
Another interesting bit, is last March AWS changed something in the control plane which both triggered a multi-day LSE, and ultimately increased attachment times from 2-3s to 10-20s (also visible in the graphs).
Lambda create-function control plane operations are still failing with InternalError for us - other services have recovered (Lambda, SNS, SQS, EFS, EBS, and CloudFront). Cloud availability is the subject of my CS grad research, I wrote a quick post summarizing the event timeline and blast radius as I've observed it from testing in multiple AWS test accounts: https://www.linkedin.com/pulse/analyzing-aws-us-east-1-outag...
For networking, the site only reports uptime % for zonal, regional, cross-region or cross-cloud tests. It excludes last mile network tests as those fail frequently due to many hops and endpoint unreliability (we use Ripe Atlas and Globalping.io endpoints which are not always reliable even with redundant probes per test).
For data and control plane I can determine the issue from the API request/response logs (i.e., network timeout, 5xx, etc.). Network tests are trickier and we don't have a great way to validate failure cause each of those events (i.e., we don't capture a traceroute on failure), other than to evaluate results from multiple endpoint combinations (e.g., AWS us-east-1 to us-west-1 fails while us-east-2 to us-west-1 succeeds).
There is lot of statistical and empirical data on this topic - MTBF estimates from vendors (typically 100k - 1m+ hours), Backblaze and Google drive failure data (~1-2% annual failure rate), IEEE and others. With N+1 redundancy (backup servers/RAID + spare drives) and proper design and change control processes, operational failures should be very rare.
With cloud hardware issues are just the start - yes you MUST "plan for failure", leveraging load balancers, auto scaling, cloudwatch, and dozens of other proprietary dials and knobs. However, you must also consider control plane, quotas, capacity, IAM, spend, and other non-hardware breaking points.
You're autoscaling isn't working - is the AZ out of capacity, did you hit a quota limit, run out of IPv4s, or was an AMI inadvertently removed? Your instance is unable to write to S3 - is the metadata service being flakey (for your IAM role), or is it due to an IAM role / S3 policy change? Your Lambda function is failing - did it hit a timeout, or exhaust the (512MB) temp storage? Need help diagnosing an issue - what is your paid support tier - submit a ticket and we'll get back to you sometime in the 24 hours.
> Any usecase that requires interacting with a service is automatically excluded
This isn't correct. Many applications consist of a mix of static and dynamic content. Even dynamic content is often cacheable for a time. All of this can be served by a CDN (using TTLs) which is a much simpler and more cost effective solution than multi-region cloud infra, with the same performance benefits.
Complex cloud infra can also fail for all kinds of reasons, and they are often harder to troubleshoot than a hardware failure. My experience with server grade hardware in a reliable colo with a good uplink is it's generally an extremely reliable combination.
If have a global user base, depending on your workload, a simple CDN in front of your hardware can often go a long ways with minimal cost and complexity.
As someone who spent 5 years attempting to get through zoning including a purely subjective “design review committee” in one of these affluent cities, I applaud this. While I agree with the desire to preserve the character and uniqueness one’s town, that pendulum has swung so far over driven by older, established nimbies who got theirs decades ago, or their offspring who inherited both homes and excessively low prop 13 tax rates locked in decades ago. The wealth disparity this whole system has created is staggering.
Sounds like an L1 or malfunctioning L2. I’d never revert to ICE in cold weather climates. Handling is great and ability to preheat (without idling) is a wonderful convenience on cold mornings.