Ask HN: Is it reasonable for a web service to go down because of high load?
10 comments
Any site that has a pre-planned launch and reasonable expectation of traffic should load test to a factor somewhere above their expectations (2x, 4x whatever).
However, it doesn't mean that you will catch all scenarios that may take down a service etc. Or even the SWAG was anywhere near right, maybe they tested to 2x their expectations but received 10x and didn't have enough capacity to support it.
Also in some cases I have seen enterprises not tell the technology people until a week or so before the date. They have a marketing team update the website but never bother to tell the infrastructure people and sometimes it is too late by the time they find out to do anything. Or the tech team is told or finds out and then is told not to spend hours on testing the site or fixing any issues, which I have seen too.
However, it doesn't mean that you will catch all scenarios that may take down a service etc. Or even the SWAG was anywhere near right, maybe they tested to 2x their expectations but received 10x and didn't have enough capacity to support it.
Also in some cases I have seen enterprises not tell the technology people until a week or so before the date. They have a marketing team update the website but never bother to tell the infrastructure people and sometimes it is too late by the time they find out to do anything. Or the tech team is told or finds out and then is told not to spend hours on testing the site or fixing any issues, which I have seen too.
Is it "ok"? Who's to say.
Does it happen? Yes. Does it happen a lot, to a variety of companies? Yes.
I think the interesting question is how. How does a company perform so spectacularly in difficult areas (product development, viral marketing, etc) and yet fail at problems that are (arguably) easily solvable for anyone who takes the time to research them?
I mean: these guys built a phone that is being compared to the flagship models of billion dollar companies. They're not morons. They're highly technical and highly competent. So why is building for web-scale such a LOW PRIORITY?
Maybe someone with experience in this kind of business could shed some light here. I'm really curious as who what their rationale might look like internally.
Does it happen? Yes. Does it happen a lot, to a variety of companies? Yes.
I think the interesting question is how. How does a company perform so spectacularly in difficult areas (product development, viral marketing, etc) and yet fail at problems that are (arguably) easily solvable for anyone who takes the time to research them?
I mean: these guys built a phone that is being compared to the flagship models of billion dollar companies. They're not morons. They're highly technical and highly competent. So why is building for web-scale such a LOW PRIORITY?
Maybe someone with experience in this kind of business could shed some light here. I'm really curious as who what their rationale might look like internally.
Free advertisement (people talk about it, and the phone is so good the website can't even handle the demand), and it doesn't matter since minions will still crawl to buy one asap.
Why spend money?
Why spend money?
If the load is caused by people actually wanting to buy a item from you like a Phone, or people wanting to use your product like Netflix, gMail etc, then downtime is a serious problem as your losing sales and/or damaging the product.
Even more so if it was a date and time publicly stated by the company, they should've put even more testing and preparation into it.
However if it's something like a side company blog which has got popular then it's not 'critical' to have it up all of the time, but they should certainly try to as it doesn't help the company image.
Even more so if it was a date and time publicly stated by the company, they should've put even more testing and preparation into it.
However if it's something like a side company blog which has got popular then it's not 'critical' to have it up all of the time, but they should certainly try to as it doesn't help the company image.
Reminds me of an interesting story. Someone I know was considering buying Google during the online IPO sale. The website was very laggy and barely worked, so he decided not to buy the stock. "If they can't handle this issue, how could they be a good web company?" Of course my friend was wrong. And it's easier now to scale than it was several years ago. But it's an interesting story. It's not always easy to anticipate what is necessary to scale or what load will be like.
I set an alarm at 4AM NZ for the OnePlus preorder, it was a huge pain in the ass. They were running Magento (!) on what appeared to be AWS and expecting Cloudflare to magically pick up all the slack. Guess they never heard of load testing (or at least a static PayPal cart as a bulletproof failsafe).
Apparently they also had a bunch of staff in the office at the time; wonder if they at least attempted to spin up a few more servers and increase the DB instance size?
Apparently they also had a bunch of staff in the office at the time; wonder if they at least attempted to spin up a few more servers and increase the DB instance size?
In current times if a site was expecting high load or large amounts of traffic, it would be a bad idea NOT to have it behind a well configured load balancer and tested to a scale beyond what you expect. A simple python web API can handle 3k requests per second on a fairly average server, so either your software choice or production method would need analysing if you are worried about downtime at high usage.
These people thought a lot about the "look" of the site, but not about the engineering or being effective at e-commerce.
For instance, they use PayPal as a payment system which is the worst choice possible in terms of user experience, particularly now when there are things like Stripe and Square.
Although the images are luscious, they take a long time to load, and if you try to read the "normal" size text, you'll find it hard because of poor contrast against the background and actually defective layout that causes text to overlap.
For instance, they use PayPal as a payment system which is the worst choice possible in terms of user experience, particularly now when there are things like Stripe and Square.
Although the images are luscious, they take a long time to load, and if you try to read the "normal" size text, you'll find it hard because of poor contrast against the background and actually defective layout that causes text to overlap.
It might be that they are running on a shoestring budget and don't have the resources to maintain a high availability web service yet.
Obviously it's not reasonable. If downtime means significant loss of revenue, then the engineers need to account for it.
This is happening now a days!!
http://supremeinv.com/
Their servers were OK just until the pre-orders started, then their whole website went down, then their login service went down, then the preorder they were showing didn't load from DB half of the time and lastly, confirming the order through PayPal gives an error.
I understand that they are under heavy load but:
1) They knew the exact time when there will be heavy load.
2) They could have done load testing.
AFAIK they also have their own servers instead of using some kind of PaaS (or IaaS) that scales instantly.
Do you think that today it is still reasonable for web services to go down because of high load? It is OK for a blog hosted on DigitalOcean to go down if it went viral unexpectedly but what about online shops that knew exactly when the spike in usage will be?