So long AWS, and thanks for all the fish: FathomDB (W08) focusing on its new DB(blog.fathomdb.com)
blog.fathomdb.com
So long AWS, and thanks for all the fish: FathomDB (W08) focusing on its new DB
http://blog.fathomdb.com/53681308
4 comments
Thanks for the good-luck wishes. Obviously it's a big undertaking, but we're seeing very promising results, so it's time to go all-in. Take that as you will :-)
Incidentally, Oracle and IBM both have clustered database products, though they can certainly be improved upon.
Incidentally, Oracle and IBM both have clustered database products, though they can certainly be improved upon.
Oracle and IBM do have clustered stuff and it's amazing, but it still tends to assume closely linked networks of high-reliability hardware and have trouble with very large, slowly linked networks of low-reliability hardware.
You might find Greenplum and VoltDB to be more interesting models to study.
The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.
You might find Greenplum and VoltDB to be more interesting models to study.
The key question is: are you aiming at OLTP or OLAP? Because, as those boring mainframe-era guys have discovered, it's difficult to serve two very different purposes.
This was the perspective that I have as well, when I was with Network Appliance we spent a lot of time dealing with folks who wanted bullet proof relational databases running on clusters, NetApp provided an extremely reliable storage platform and a unique ability to 'snapshot' instantly the current state of the storage used by the data base so that you could run various experiments against that data without huge copies etc.
There were multiply redundant switches and massive investments in transactional data recovery. The big finance guys were particularly intimidating when they talked about several hundred billion dollars in transactions in a 24 hour period, that is $4,000 - $5,000 per mS, or 4 to 5 million dollars per second. Combined with the whole finance game of not confirming receipt until the last possible moment to keep your liabilities managable, ouch.
I would love to see a relational database emerge that could do ACID on an unreliable cloud, and I think the folks who achieve that should become gazillionaires. I was just noting that a whole lot of money and research has already been poured into that hole. We're still waiting to here the 'thump' of it hitting the bottom. :-)
It totally rocks that Fathom is taking on that challenge.
There were multiply redundant switches and massive investments in transactional data recovery. The big finance guys were particularly intimidating when they talked about several hundred billion dollars in transactions in a 24 hour period, that is $4,000 - $5,000 per mS, or 4 to 5 million dollars per second. Combined with the whole finance game of not confirming receipt until the last possible moment to keep your liabilities managable, ouch.
I would love to see a relational database emerge that could do ACID on an unreliable cloud, and I think the folks who achieve that should become gazillionaires. I was just noting that a whole lot of money and research has already been poured into that hole. We're still waiting to here the 'thump' of it hitting the bottom. :-)
It totally rocks that Fathom is taking on that challenge.
OLTP. I personally believe Hadoop has got the OLAP problem-space covered.
What are your most important non-functional requirements?
In particular, how do you see the tradeoff between durability and performance?
In particular, how do you see the tradeoff between durability and performance?
You've totally hit the nail on the head there. Up until a month ago, it was a fairly obsessive compulsion around performance. Durability and reliability are features of the design, so it was about getting performance under the design.
However, the AWS outage has (we hope) put reliability back on the map. So maybe now we don't have to be the fastest database, if we're the fastest reliable database. That's a liberating change of perspective, which has been behind many of the recent advances.
However, the AWS outage has (we hope) put reliability back on the map. So maybe now we don't have to be the fastest database, if we're the fastest reliable database. That's a liberating change of perspective, which has been behind many of the recent advances.
Does durability absolutely trump performance, or are you prepared to trade off?
For example, if durability is an absolute must (as it is for current databases), then your main bottlenecks would be a) spinning rust and b) network traffic to distribute copies of the data throughout the network. SSDs will help this a lot, but already you're back where almost everyone has already started.
If you're prepared to accept highly replicated in-memory copies as "durable", you can already make your overall system hundreds or even thousands of times faster.
So how do you define durability?
For example, if durability is an absolute must (as it is for current databases), then your main bottlenecks would be a) spinning rust and b) network traffic to distribute copies of the data throughout the network. SSDs will help this a lot, but already you're back where almost everyone has already started.
If you're prepared to accept highly replicated in-memory copies as "durable", you can already make your overall system hundreds or even thousands of times faster.
So how do you define durability?
1 copy on 1 disk?
x copies on y disks in the same computer?
x copies in y disks in different computers?
x copies in RAM of y different computers?I agree that it reads poorly. If I was a tinier bit more motivated to finish my HN session more quickly, I would've assumed that FathomDB did lose some of their customers' data. That's what I thought when I read the headline, and only when I started reading the article did I see that they didn't lose any of their customers' data.
sorry, this is probably tangential, but how is the AWS outage any different from any other hardware outage, regardless of whether or not you actually own the hardware?
why does it seem that everyone expects 100% uptime from a VM just because it's in "the cloud"? shouldn't "the cloud" be used to make fault-tolerance even easier, because you have access to multiple geographic regions and multiple providers with little fuss?
they're still computers, and they're still bound to go down occasionally. i don't see how running leased VMs somehow absolves you of doing basic operations work and guarantees a bulletproof experience. regardless of where you host it, writing a nicely distributed and fault-tolerant system is and always has been difficult. the only part that's gotten easier is finding rack space.
why does it seem that everyone expects 100% uptime from a VM just because it's in "the cloud"? shouldn't "the cloud" be used to make fault-tolerance even easier, because you have access to multiple geographic regions and multiple providers with little fuss?
they're still computers, and they're still bound to go down occasionally. i don't see how running leased VMs somehow absolves you of doing basic operations work and guarantees a bulletproof experience. regardless of where you host it, writing a nicely distributed and fault-tolerant system is and always has been difficult. the only part that's gotten easier is finding rack space.
With physical hardware, the basic failure characteristics etc are well known. When the hardware is virtual or abstracted as it is on the cloud, you have no choice but to rely on the promises made by the provider. If you can't build a reliable system on the abstraction provided by the cloud, you can't reliably use the cloud. But with a good cloud, you can architect a reasonable solution based on the provider's promises.
However, if the provider doesn't keep those promises, then all your hard work and calculations go totally out the window.
At that point, you have to figure out a way to run a database on a system that effectively offers you no guarantees. That's what we're working on.
However, if the provider doesn't keep those promises, then all your hard work and calculations go totally out the window.
At that point, you have to figure out a way to run a database on a system that effectively offers you no guarantees. That's what we're working on.
For your application though a failure is a failure in any scenario. EBS went down in one zone and was slow in a couple others for a few hours, it's a bad downtime but some services survived the problem relatively unscathed. This suggests that something could have been done to avoid any trouble at all.
If the datacenter you are in has a slight conditioning problem this summer and 10% of your drives breaks down due to excessive heat, how quickly will you be able to re-provision the data center?
About a week after AWS outage the italian ISP Aruba had a UPS failure due to a fire in the UPS room and the entire datacenter switched off automatically during the night. For the following 8 hours that datacenter was off for every customer. How would your new solution handle such a situation?
Designing a hot standby replication solution for MySQL/PostgreSQL that works across regions seems easier to me rather than implementing a database from scratch that should solve a very complex problem.
If the datacenter you are in has a slight conditioning problem this summer and 10% of your drives breaks down due to excessive heat, how quickly will you be able to re-provision the data center?
About a week after AWS outage the italian ISP Aruba had a UPS failure due to a fire in the UPS room and the entire datacenter switched off automatically during the night. For the following 8 hours that datacenter was off for every customer. How would your new solution handle such a situation?
Designing a hot standby replication solution for MySQL/PostgreSQL that works across regions seems easier to me rather than implementing a database from scratch that should solve a very complex problem.
Certainly it is possible to engineer a hot standby solution for MySQL databases using DRBD, or synchronous replication etc. There's a choice between engineering an endless series of fixes like that, or re-examining the problem space and building a new database. After years of doing the former, we chose to examine the latter, and we're seeing good indications from that approach.
I have misgivings because you are not the first to try and re-examine the problem space. A lot of the endless series of fixes have arisen from just such attempts.
I want you to succeed, but you're dealing with seriously hairy deep magic issues that the best minds in the industry and academia have been chipping away at for decades.
I want you to succeed, but you're dealing with seriously hairy deep magic issues that the best minds in the industry and academia have been chipping away at for decades.
A healthy skepticism, then: nothing wrong with that. It's a speculative project with a big payoff if it succeeds, and I wouldn't ask you to believe until we've shown a working product. Your good wishes are appreciated though!
sorry but that seems really naive. an SLA is not a guarantee of service -- you can't force a computer or network to stay online because someone with a sheet of paper said so. it's a target for a best-effort guarantee and when that guarantee is broken then you get a bit of a refund.
it's like when you have a big building project and there's a provision that the contractor will refund $5000/day for every day past march 1st. doesn't mean march 1st never gets passed.
failure modes in the cloud are known too: your VMs are either working or they aren't, or maybe they're somehow degraded but you should fail over anyway. it's very similar to physical hardware. what will you think when a backhoe takes out your datacenter's fiber for 10 hours? that physical hardware and datacenters are now unreliable too?
as for a database that runs on a system that offers no guarantees -- isn't that all of them?
it's like when you have a big building project and there's a provision that the contractor will refund $5000/day for every day past march 1st. doesn't mean march 1st never gets passed.
failure modes in the cloud are known too: your VMs are either working or they aren't, or maybe they're somehow degraded but you should fail over anyway. it's very similar to physical hardware. what will you think when a backhoe takes out your datacenter's fiber for 10 hours? that physical hardware and datacenters are now unreliable too?
as for a database that runs on a system that offers no guarantees -- isn't that all of them?
Totally agree that SLAs are not a guarantee of service - I don't believe I said they were, and I was trying to make the exact same point: too many people treat them as if they were a guarantee, even when they carry only token penalty clauses.
Separately from the SLA, technical promises/guarantees that AWS did make e.g. isolated AZs were broken in the April outage.
I think that your proposed model ("machine is online or not") may be sufficiently simple that the AWS cloud can satisfy it; however I think it is very difficult to build anything interesting if that is the only axiom you have. In particular, I would want something related to persistent storage in the model, or else storing state becomes very difficult.
Separately from the SLA, technical promises/guarantees that AWS did make e.g. isolated AZs were broken in the April outage.
I think that your proposed model ("machine is online or not") may be sufficiently simple that the AWS cloud can satisfy it; however I think it is very difficult to build anything interesting if that is the only axiom you have. In particular, I would want something related to persistent storage in the model, or else storing state becomes very difficult.
This author blames Amazon for a poor SLA yet continues using their service?
What cloud provider offers a more reasonable SLA?
What cloud provider offers a more reasonable SLA?
I'm the author. I think you're conflating the SLA with the guarantees the cloud offers: one of the points I tried to make was that an SLA doesn't offer you any real guarantees; they only really benefit the service provider. If every SLA gave you a "50 times" penalty, as FathomDB is paying, then they would be interesting, because at that level the provider has to take uptime seriously.
In terms of who offers a better cloud, clouds based on OpenStack are 'the great hope'. IaaS is about commoditizing infrastructure, so open source seems necessary; rather than AWS's very closed, secretive approach.
In terms of who offers a better cloud, clouds based on OpenStack are 'the great hope'. IaaS is about commoditizing infrastructure, so open source seems necessary; rather than AWS's very closed, secretive approach.
I deal with SLAs in my job all the time. After over a decade negotiating uptime SLAs, I'm firmly convinced that the best protection against downtime is a huge, vocal userbase, and the best remedy for downtime is an unqualified termination right and refund of prepaid fees.
An SLA (even with severe penalties) is certainly setting up the right incentives on paper, but in practice it has little effect on the results. Why? Because enforcing a promise on a piece of paper is bothersome, expensive and sours the relationship.
Also, saying you'll pay 50x penalty is only half the story. What are the triggering events that entitle customers to payment? Is the SLA measured end-to-end (i.e., from the user's endpoint to yours)? Is the SLA penalty cumulative? Can your customers actually profit from your downtime (i.e., you would end up owing them more cash than they paid you)?
Without more details, your post suggests that each fraction of a second of downtime experienced by any single user of your service entitles them to a credit of 50x what they paid for that fraction of a second. Is that your policy?
An SLA (even with severe penalties) is certainly setting up the right incentives on paper, but in practice it has little effect on the results. Why? Because enforcing a promise on a piece of paper is bothersome, expensive and sours the relationship.
Also, saying you'll pay 50x penalty is only half the story. What are the triggering events that entitle customers to payment? Is the SLA measured end-to-end (i.e., from the user's endpoint to yours)? Is the SLA penalty cumulative? Can your customers actually profit from your downtime (i.e., you would end up owing them more cash than they paid you)?
Without more details, your post suggests that each fraction of a second of downtime experienced by any single user of your service entitles them to a credit of 50x what they paid for that fraction of a second. Is that your policy?
Thanks for the expert input. This is exactly why we don't have a SLA at FathomDB, because I don't think that they're worth the paper they're written on. It seems like you almost agree (?)
Most cloud providers (including FathomDB) charge by the hour anyway, so you have termination rights for any reason, and there are no prepayments to refund.
We think that 'doing the right thing' is the only workable policy, and it shouldn't require an outcry from a vocal userbase. We should have handled the outage better, and we're making a big payment to demonstrate that. We're not going to let any new customers use a system where we can't be confident in its uptime (MySQL on AWS); we're building a new system in which we can be confident.
Most cloud providers (including FathomDB) charge by the hour anyway, so you have termination rights for any reason, and there are no prepayments to refund.
We think that 'doing the right thing' is the only workable policy, and it shouldn't require an outcry from a vocal userbase. We should have handled the outage better, and we're making a big payment to demonstrate that. We're not going to let any new customers use a system where we can't be confident in its uptime (MySQL on AWS); we're building a new system in which we can be confident.
Dead on. I went back and re-read the article, and yes – we are in agreement. It's about doing whatever it takes to keep the relationship with your customers on an even keel. I couldn't agree more that SLAs are generally bad for customers, even though the perception is the opposite. I can't tell you how many times I've had a client demand an SLA from their vendor, and when that SLA arrives, it reads like a commitment, but actually it's impossible to fail. And these almost always are wrapped in "exclusive remedy" language, which quite literally means the customer would be better off without it. The devil is in the details.
Cheers!
Cheers!
I should have rephrased better; "What provider offers a SLA that takes IaaS uptime more seriously than Amazon?"
Can you actually tell us a bit about the new DB you're building? (What is it designed to run on, what kind of failures are you aiming to tolerate, and so on.)
Sorry, not too many details yet - let us finish it first :-) Commodity hardware though, and quorum-style failure-tolerance characteristics.
FathomDB customers saw outages because the implementation Fathom did using MySQL on AWS went out when AWS went out.
So in conclusion Fathom is going to build something which Oracle, IBM, SyBase, and every other database company in the history of the planet has tried and failed to do, build a relational database on a distributed infrastucture that is more reliable than the underlying infrastructure? Uh, good luck with that. Seriously, its Turing Prize material if you succeed.
I was thinking the punch line would be "Gee Netflix uses AWS and they didn't go down so we're going to more what they did." I guess they are going to compete with MongoDB and Riak.