Amazon’s Scorpion Problem(feld.com)
feld.com
Amazon’s Scorpion Problem
http://www.feld.com/archives/2014/07/amazons-scorpion-problem.html
13 comments
Especially #3. Amazon is willing to let themselves be scammed before denying free replacement products to customers.
I've had very good customer service purchasing physical items from amazon.
ec2 and associated services' customer service and support has been pretty shitty, and these are from companies paying anywhere from $8k to $96k/mo
ec2 and associated services' customer service and support has been pretty shitty, and these are from companies paying anywhere from $8k to $96k/mo
I agree that their AWS support needs a lot of improvement, and there are others in the industry (rackspace, peer one) who are better. But I disagree with the OP's assertion that Amazon is inherently incapable of providing good customer service in any of its markets.
That was exactly my point :)
That isn't true. I've had to issue chargebacks when Amazon's vendor and Amazon couldn't come to an understanding and I never received the software product w/o a revoked key.
Might also be country dependent. I'm using amazon.de (besides .com for prime instant) and Germany is known for it's strong consumer protection laws so Amazon might have beefed up their service here. But I've read some articles before from US customers having amazing experiences.
edit: Sorry, didn't see "vendor" before. True, when you are using marketplace you seem to be pretty much on your own.
edit: Sorry, didn't see "vendor" before. True, when you are using marketplace you seem to be pretty much on your own.
Ah, but it was Amazon selling the product, not the vendor. The vendor claims Amazon revoked the keys so when you try to activate the product it was unusable without using a crack.
#2 always was what impressed me most about AWS. When I was trying out Digital Ocean, I had to hack together my own scripts to set up the equivalent of cloudfront, auto-scaling, and elastic load balancing. I realized just how important those seemingly small infrastructure tools are when they "just work".
Exactly. The value of AWS is not in the individual pieces, but in how all the pieces fit together, and how a novice is able to do it.
Regarding #4, do you know how common it is to spend over $200k per month. If it is a rare use case I would not be surprised if those companies are making a sub-optimal decision. In my anecdotal experience, whenever we considered that order of magnitude of spend on AWS we ended up building our own infrastructure.
"AWS is not the best product at anything – most of their features are mediocre knock offs of other products."
Does not list a single example.
Does not list a single example.
Elastic Transcoder is a good example. They don't even support variable bitrate, so you have to choose between huge files and video that looks good for a while until you hit some dark scenes or high contrast sequences that exhibit terrible artifacting.
Anyone who makes the blanket assertion that "AWS is not the best product at anything" is failing to grasp the fundamental advantage of AWS--it is the only (and therefore best and cheapest) "full line" IAAS provider. Obviously simple value added services like Dropbox that primarily leverage just one piece of the cloud (storage) will eventually move to wherever (and everywhere) that they can get their infrastructure cheapest. When complex services like Netflix begin moving off of AWS is when I'd start worrying.
The other thing that AWS has going for it is widely adopted third party support. Pretty much any software, site, tool or service that has any sort of cloud monitoring, deployment etc feature will be integrated with AWS. This not true for any of AWS's competitors. I've never seen a third-party provide support for an AWS competitor before supporting AWS. I know monitoring tools that do not support Azure, Google Cloud or Rackspace. I know libraries that don't support all of them but always support AWS.
In addition most sysadmins and developers will be more likely familiar with AWS's tools, API and dashboard. They'll be more likely to already have their development environment set up with AWS command lines tools. You ever heard of someone with HP cloud tools installed [1]? I frequently browse dotfiles on GitHub and often see non-sensitive AWS credentials and aliases. I've never seen a Rackspace or Google Cloud thing in them.
AWS wins on mindshare. It's also a proven system. Infrasture decisions are really important, and downtime is very bad and when you choose AWS you know at the very least that if your shit goes down because of provider issues everyone else will be down too and you can kind of get a pass (although outsourcing downtime is kind of shitty), but if you're not on AWS and your shit goes down because of the provider people will ask you why aren't you using AWS. Your customers and investors might actually be unhappy with you because of this.
As for support Google also has terrible support, Rackspace's managed support is very expensive and actually probably not what a competent team would want, otherwise it is easy to get a person on the phone at Rackspace, so that is commendable. Support doesn't scale, but if you are paying AWS a lot I bet you can get support.
I don't really like the mono-culture of AWS everywhere, but I feel like the points above need to be raised. A standard on APIs like OpenStack would be nice, but implementations will differ and AWS will fight this kind of change and because of their dominance getting traction will be hard.
[1] this is actually a thing: https://docs.hpcloud.com/cli
In addition most sysadmins and developers will be more likely familiar with AWS's tools, API and dashboard. They'll be more likely to already have their development environment set up with AWS command lines tools. You ever heard of someone with HP cloud tools installed [1]? I frequently browse dotfiles on GitHub and often see non-sensitive AWS credentials and aliases. I've never seen a Rackspace or Google Cloud thing in them.
AWS wins on mindshare. It's also a proven system. Infrasture decisions are really important, and downtime is very bad and when you choose AWS you know at the very least that if your shit goes down because of provider issues everyone else will be down too and you can kind of get a pass (although outsourcing downtime is kind of shitty), but if you're not on AWS and your shit goes down because of the provider people will ask you why aren't you using AWS. Your customers and investors might actually be unhappy with you because of this.
As for support Google also has terrible support, Rackspace's managed support is very expensive and actually probably not what a competent team would want, otherwise it is easy to get a person on the phone at Rackspace, so that is commendable. Support doesn't scale, but if you are paying AWS a lot I bet you can get support.
I don't really like the mono-culture of AWS everywhere, but I feel like the points above need to be raised. A standard on APIs like OpenStack would be nice, but implementations will differ and AWS will fight this kind of change and because of their dominance getting traction will be hard.
[1] this is actually a thing: https://docs.hpcloud.com/cli
Eh? What do you consider "full line"?
I'm pretty sure Netflix uses it due to the highly variable usage situation. It never makes sense to have a server sitting idle for 23 hours a day.
I'm pretty sure Netflix uses it due to the highly variable usage situation. It never makes sense to have a server sitting idle for 23 hours a day.
I think "full line" isn't an entirely static concept in IAAS, but I'm using it here to suggest there's no other player offering the same breadth of services. I haven't surveyed other providers in detail lately, but I expect for any given competitor it wouldn't take you long to find some AWS service (http://aws.amazon.com/products/) that they simply don't have an equivalent for.
EDIT: found someone who's made a feature grid here: http://www.planforcloud.com/pages/resources/cloud_services.h...
This doesn't even include more esoteric stuff like Redshift and Elastic Beanstalk.
EDIT: found someone who's made a feature grid here: http://www.planforcloud.com/pages/resources/cloud_services.h...
This doesn't even include more esoteric stuff like Redshift and Elastic Beanstalk.
I'm amused by the downvotes combined with the unwillingness to define "full line".
> AWS is unbelievably lousy at support.
Compared to whom? Google? How many times have you seen tickets like [1] for Android, Gapps, etc? Additionally, the google cloud platform onboarding was so bad, I almost gave up a $500 credit in frustration.
[1] https://productforums.google.com/forum/#!topic/gmail/HTX9cLb...
Compared to whom? Google? How many times have you seen tickets like [1] for Android, Gapps, etc? Additionally, the google cloud platform onboarding was so bad, I almost gave up a $500 credit in frustration.
[1] https://productforums.google.com/forum/#!topic/gmail/HTX9cLb...
I also received the free $500 credit and decided not to use it because of an on boarding issue.
On the other hand, when Amazon first released EC2, they gave me a $1000 credit to use anyway I wanted, and that I did use.
I really like Google products like GMail, Blogger, and G+ and working at Google the infrastructure was more than wonderful, but for external developers the Google Cloud Platform products really could be a lot better.
On the other hand, when Amazon first released EC2, they gave me a $1000 credit to use anyway I wanted, and that I did use.
I really like Google products like GMail, Blogger, and G+ and working at Google the infrastructure was more than wonderful, but for external developers the Google Cloud Platform products really could be a lot better.
Rackspace leaps to mind.
Rackspace's support is indeed fantastic, however their product is significantly more friction-ful to use (I'm currently at a company that has purchased a Rackspace Private Cloud, and uses it in conjunction with their public cloud). It works as advertised, kinda-sorta and with a whole lot of spit, duct-tape, and prayer, but I'd much rather use Amazon's system and deal with one support ticket every other quarter, than the weekly support ticket(s) using Rackspace's product seems to require.
I call bullshit on the support and the "knockoff" claim. I've always felt they are the best with support - they've even waived an accidental $120 instance for me. Yhe tools they provide me are exemplary and cannot be replaced IMO.
What does this have to do with "scorpion"?
From the title, I thought it would be about Amazon competing with its customers, like Netflix using AWS to compete with Amazon Prime, and how its inevitable that Amazon will eventually have to sting Netflix.
From the title, I thought it would be about Amazon competing with its customers, like Netflix using AWS to compete with Amazon Prime, and how its inevitable that Amazon will eventually have to sting Netflix.
I also was confused but I think he was referring to the constant (alleged) behavior of Amazon towards his portfolio companies in which they act like partners/potential acquirers and then just build the tech themselves.
From the article: "In addition, they always end up competing on every front possible, hence the chatter about Dropbox moving away from AWS since AWS has now come out with a competitive product. It appears that it’s just not in Amazon’s nature to collaborate with others."
$200k that is insane, it is cheaper to go width own hardware long before that.
The server room we have at our office have had max 5 hours of total down time since 2011 And we have moved our entire office under that period. I will try to estimate our general costs in EUR.
Inital costs for servers, switches, cooling, Racks UPS 150K
Replacement of servers every 3-4 years 60K/3 = 20K
Internet line = 10K a year
Electricity = 2K a year
Consulting cost for hardware maintenance = 10K
Old servers we put those in an a room a bit away and use them for backups/failsafe.
That is about 50K a year.
Counting in initial costs and just cause is is till almost nothing compared to 200K a month, I would say that switching to self hosting is financially beneficial when getting to about 10K a month with 3party hosting.
The server room we have at our office have had max 5 hours of total down time since 2011 And we have moved our entire office under that period. I will try to estimate our general costs in EUR.
Inital costs for servers, switches, cooling, Racks UPS 150K
Replacement of servers every 3-4 years 60K/3 = 20K
Internet line = 10K a year
Electricity = 2K a year
Consulting cost for hardware maintenance = 10K
Old servers we put those in an a room a bit away and use them for backups/failsafe.
That is about 50K a year.
Counting in initial costs and just cause is is till almost nothing compared to 200K a month, I would say that switching to self hosting is financially beneficial when getting to about 10K a month with 3party hosting.
Without numbers showing the actual quantities involved this cost comparison is kind of useless. Not that I doubt going with your own hardware is cheaper at those levels, it's just not a conclusion I can draw from what you've written. Also, as others said, labor costs need to be factored in.
How much would a comparable setup cost on AWS per year? Once we know that we can draw conclusions.
How much would a comparable setup cost on AWS per year? Once we know that we can draw conclusions.
We did calculations on that about 2 years ago, then it was about 700K pear year.
But that left out some cost like Shared network folders that our users use, so we would have needed to get some kind of replacement for our netapp(a form as NAS).
But that left out some cost like Shared network folders that our users use, so we would have needed to get some kind of replacement for our netapp(a form as NAS).
Where are your costs for labor. Is that all included in your consulting cost? If we as an advertising startup decided to migrate all that internally, I'm sure our costs would go up not only in manpower but also from the distraction of dealing with that rather than focusing on our product.
The initial cost was with labor included.
The labor is the consulting cost.
But to be honest there is not like one has to do much width the hardware.
When one hardware server fails all the virtual machines will still just continue on another hardware server. Vmware takes care of that stuff and it just works, at least in our testing of pulling the cords to the server (stuff has not failed yeat).
Well if money is no object, you are definitely doing the right thing.
But to be honest there is not like one has to do much width the hardware.
When one hardware server fails all the virtual machines will still just continue on another hardware server. Vmware takes care of that stuff and it just works, at least in our testing of pulling the cords to the server (stuff has not failed yeat).
Well if money is no object, you are definitely doing the right thing.
I respect Brad and usually enjoy his posts, but this one is full of s...
He doesn't give a single example of why "AWS is not the best product at anything". This is obviously true, but the same could be said about Google, Microsoft, or Rackspace. So what's the point? Who has a better offering?
The discussion of AWS being more expensive than bare metal is valid, but that's an obvious statement. Amazon uses bare metal (albeit at scale), and adds their application stack + support. You're paying for the convenience of purchasing all services integrated from a single vendor. If your business is simple enough and you can just throw hardware and bandwidth to solve your needs, then bare metal will always be cheaper. But for more complex environments (e.g., multi-regions, global distribution, auto-scaling), or when you need to quickly scale up and down (to accomodate surge in demand, for example) bare metal is not an option.
Net net, this sounded like a rant against Amazon's world domination plans, and the threat to the very existence of some of the companies on his fund. He said it best:
"We used to think of Amazon as a potential acquirer for these companies, or at least a powerful strategic partner. Now we know they are just using the bait of “we want to work more closely with you” as market and product intelligence."
If your business model depends on a strategic partnerships with or being acquired by Amazon, you're doing it wrong.
He doesn't give a single example of why "AWS is not the best product at anything". This is obviously true, but the same could be said about Google, Microsoft, or Rackspace. So what's the point? Who has a better offering?
The discussion of AWS being more expensive than bare metal is valid, but that's an obvious statement. Amazon uses bare metal (albeit at scale), and adds their application stack + support. You're paying for the convenience of purchasing all services integrated from a single vendor. If your business is simple enough and you can just throw hardware and bandwidth to solve your needs, then bare metal will always be cheaper. But for more complex environments (e.g., multi-regions, global distribution, auto-scaling), or when you need to quickly scale up and down (to accomodate surge in demand, for example) bare metal is not an option.
Net net, this sounded like a rant against Amazon's world domination plans, and the threat to the very existence of some of the companies on his fund. He said it best:
"We used to think of Amazon as a potential acquirer for these companies, or at least a powerful strategic partner. Now we know they are just using the bait of “we want to work more closely with you” as market and product intelligence."
If your business model depends on a strategic partnerships with or being acquired by Amazon, you're doing it wrong.
That was a strange article.
Isn't it a self obvious truth that AWS is most appropriate when you need lots of computational resources in bursts?
If you often need many extra servers for a short period of time, then AWS is really good for that. Another thing that AWS provides is some very handy infrastructure APIs like S3, DynamoDB, Eleastic Mapreduce, etc.
For flat usage scenarious, then renting beefy VPSs or physical servers is more cost effective.
Isn't it a self obvious truth that AWS is most appropriate when you need lots of computational resources in bursts?
If you often need many extra servers for a short period of time, then AWS is really good for that. Another thing that AWS provides is some very handy infrastructure APIs like S3, DynamoDB, Eleastic Mapreduce, etc.
For flat usage scenarious, then renting beefy VPSs or physical servers is more cost effective.
where do people get these numbers? You should move to physical hardware well before $200k/month.
They probably assume you have to pay for 24/7 DC tech / sysadmin types to do the racking, setup, etc. that AWS automates and are factoring it into the price.
To be fair, that is part of the cost. However, unless you are really big, you will just pay remote hands which is like $100-200/hr. The number of incidents are probably way too few, even at 200k a month, for 24/7 on-site sysadmins/techs.
To be fair, that is part of the cost. However, unless you are really big, you will just pay remote hands which is like $100-200/hr. The number of incidents are probably way too few, even at 200k a month, for 24/7 on-site sysadmins/techs.
Do you have a source or some insight you could share as to when a company should move away from AWS? A quick google search turns up $1000 per square foot[1] for a buidling a datacenter. There's also a nice discussion on reddit[2], but there's no talk of cost. I think this is an interesting economics? problem and would like to hear more on it if anyone has the time.
[1]http://www.webhostingtalk.com/showthread.php?t=920078 [2]http://redd.it/16m9pq
[1]http://www.webhostingtalk.com/showthread.php?t=920078 [2]http://redd.it/16m9pq
I can tell you that you can rent an entire rack with power for $500-$1000/month (42U);
A decent internet connection from level3 or the colo house internet for another few hundred.
So even at $2k/month we've still got $180k/month to buy servers and storage (or $2M for the year).
$200k/month is a lot of money.
A decent internet connection from level3 or the colo house internet for another few hundred.
So even at $2k/month we've still got $180k/month to buy servers and storage (or $2M for the year).
$200k/month is a lot of money.
What the crap? Look what you are saying is crazy.
1) you aren't including operations staff.
2) what are you doing for DR/HA?
3) how about CDN etc?
4) how about load balancing between datacenters? DNS failover?
5) how many units of computer would $200k be at amazon? I'd imagine a lot. I recently (nearly a year ago) specced out 16 servers on AWS with HA/DR on opsworks for $15k over 3 years ($5k/year). Back of the envelope calculations make that 640 cores for your $200k. With 400GB transfer a month.
6) what are you doing for backups?
7) what if you don't need to utilize a whole rack? Most of the time systems don't fully utilize racks. But you're paying for them anyway.
1) you aren't including operations staff.
2) what are you doing for DR/HA?
3) how about CDN etc?
4) how about load balancing between datacenters? DNS failover?
5) how many units of computer would $200k be at amazon? I'd imagine a lot. I recently (nearly a year ago) specced out 16 servers on AWS with HA/DR on opsworks for $15k over 3 years ($5k/year). Back of the envelope calculations make that 640 cores for your $200k. With 400GB transfer a month.
6) what are you doing for backups?
7) what if you don't need to utilize a whole rack? Most of the time systems don't fully utilize racks. But you're paying for them anyway.
I think you should keep using the cloud but I'll respond to your points anyway.
1- you still need staff to deal with aws. I would argue that you need more.
2- HA is the same; DR is different. Use AWS for your DR ;)
3- What about them? there is literally no difference
4- see 3
5- a lot, I'm sure. Enough to justify owned physical equipment that you aren't sharing and doesn't get "retired" every other week
6- whatever I want, including putting them on glacier or sending them to my data center's cheap tape storage
7- it's a trivial part of the cost, why worry about it? or rent half a rack if you don't mind taking on a little extra cord-pulling risk
1- you still need staff to deal with aws. I would argue that you need more.
2- HA is the same; DR is different. Use AWS for your DR ;)
3- What about them? there is literally no difference
4- see 3
5- a lot, I'm sure. Enough to justify owned physical equipment that you aren't sharing and doesn't get "retired" every other week
6- whatever I want, including putting them on glacier or sending them to my data center's cheap tape storage
7- it's a trivial part of the cost, why worry about it? or rent half a rack if you don't mind taking on a little extra cord-pulling risk
What if your usage isn't evenly distributed across a month?
Chances are that your colo datacenter is building out an openstack-based cloud offering.
So you build your colo'd private cloud on openstack, and then use their's to handle surges. As your traffic grows, you add your own gear to handle the base line.
But ~200K per month, you're operating at a very different scale from 2k or even 20k. Having a couple of idle servers 23 hours a day is not wasteful at that scale, unless your CIO is Ebeneezer Scrooge.
So you build your colo'd private cloud on openstack, and then use their's to handle surges. As your traffic grows, you add your own gear to handle the base line.
But ~200K per month, you're operating at a very different scale from 2k or even 20k. Having a couple of idle servers 23 hours a day is not wasteful at that scale, unless your CIO is Ebeneezer Scrooge.
Then you rent out excess capacity in some sort of IAAS business model like, oh wait... AWS!
aws is a good fit if you have big spikes and massive lulls.. maybe.
It is, but I can conceive of large companies paying it because they don't also want to employ, in-house, the people necessary to maintain the physical infrastructure.
you also need people to babysit, manage, and build out ec2 instances. They are constantly failing, being retired, getting random poor performance, etcetcetc.
The failure rates vs physical hardware are incredibly high (just personal experience but I doubt you'll find many who disagree).
Once your systems are up and running you don't need to do much. Remote hands will replace a hard drive for $100, for example.
The failure rates vs physical hardware are incredibly high (just personal experience but I doubt you'll find many who disagree).
Once your systems are up and running you don't need to do much. Remote hands will replace a hard drive for $100, for example.
Yeah. You can get managed hardware (for example at Soft Layer) for a much better price/performance ratio than EC2. The upfront costs are higher but the price for equivalent hardware is way cheaper.
I think EC2 makes total sense for variable load (I paid like $0.25 to run some Coursera homework on a huge computer for 15 minutes and that was amazing) but if you're running something that needs to be up constantly and reliably (e.g., a database) EC2 gets expensive pretty fast.
I think EC2 makes total sense for variable load (I paid like $0.25 to run some Coursera homework on a huge computer for 15 minutes and that was amazing) but if you're running something that needs to be up constantly and reliably (e.g., a database) EC2 gets expensive pretty fast.
I'm not the person that did this analysis, but if I recall, for file storage, somewhere just shy of $30k/mo was the magical point at which it made financial sense to move from S3 to a few stout SoftLayer storage machines at my company.
We (Orando Labs, https://orandolabs.com) feel the sting of the scorpion. Check out EnduroSync vs Cognito at http://orandolabs.wordpress.com/2014/07/12/amazon-cognito/
Does Google also have the "lousy at support" problem? Their DNA has always favored algorithmic approaches that scale vs. human contact which doesn't. I remember hearing horror stories when they first came on the scene.
I actually find that google has quite good support (where they have support) Google Apps for business for example.
"While we are in the middle of a massive secular shift from owned data centers to outsourced data centers"
A secular shift? As opposed to a religious one?
A secular shift? As opposed to a religious one?
Like many words, secular has other senses. Per a representative definition like…
http://www.merriam-webster.com/dictionary/secular
…Feld is using the sense (3), and especially (3)(c), a quite common usage in economics (as a contrast with transient/cyclical changes).
You might benefit from a more catholic appreciation of vocabulary. (And no, that has nothing to do with the Catholic church.)
http://www.merriam-webster.com/dictionary/secular
…Feld is using the sense (3), and especially (3)(c), a quite common usage in economics (as a contrast with transient/cyclical changes).
You might benefit from a more catholic appreciation of vocabulary. (And no, that has nothing to do with the Catholic church.)
Secular also has a meaning of once in a long time or once in an age. One source I looked at says that the Latin root in also the root for "century" in Romance languages. I could argue it doesn't add much to that sentence (which I don't really agree with in any case) but it's not an incorrect use of the word.
http://en.wikipedia.org/wiki/In_saecula_saeculorum : "The saeculum in Roman antiquity was the potential maximal human lifespan, or roughly a century"
[deleted]
#1 - "AWS is not the low price provider", "Being low priced is in Amazon’s nature so this will be intensely challenging to them". This is true, but as the market-leader, do they need to be?
#2 - "most of their features are mediocre knock offs of other products". As @ToddMathews said here (https://news.ycombinator.com/item?id=8104364), he doesn't list any examples of these superior cloud services. I can't think of any. Sure, ELB is a knock off of HAProxy, but in that sense, Gmail is a knockoff of Postfix.
#3 - "AWS is unbelievably lousy at support", "Support is just not in Amazon’s nature". Really? Compared to whom? In my experience, they have always been one of the best providers of support of any online company.
#4 - "Once you are at $200k / month of spend, it’s cheaper and much more effective to build your own infrastructure". And yet many businesses spend more than $200k PCM. Are they stupid? Or is the author wrong?