Ask HN: Anyone running a free app want to try a in-ram MySQL NDB server on me?
1 comments
I am very interested in your project and I hope you publish your results. I don't have a suitable database project, unfortunately.
I have been experimenting with master-slave replication in MySQL. I am interested in performance, but redundancy and high availability are more important to me.
I have a question -- if you are doing high end RAM based MySQL servers, won't running them in a virtual machine introduce other issues ? Shouldn't these be standalone real computers ?
I have been experimenting with master-slave replication in MySQL. I am interested in performance, but redundancy and high availability are more important to me.
I have a question -- if you are doing high end RAM based MySQL servers, won't running them in a virtual machine introduce other issues ? Shouldn't these be standalone real computers ?
also, what's high-end about an in-ram database? It's all cheap commodity parts; my cost for a full on 8 core box with 32GiB ram is well under $1500, and around two amps of (American 120v) power. This is decidedly low-end, until you get, you know, a few marketing guys and 3 vars in the mix and start paying double for hardware put together by people who don't believe that ESD is a problem.
It's awesome that you can get servers like that for those prices these days. One of my customers recently purchased a similarly configured PowerEdge.
What is high-end about the database is you are trying to get high-end performance. Having all the data in RAM is the "close to the hardware" part of getting that performance. Having multiple computers is the parallelism part of that performance. Having virutualization sticks a software layer between MySQL and the RAM, and if you have other VMs running on the hardware, then any performance numbers will be dependent on what the other VMs are doing. Of course, if you have all the NDB nodes running as VMs on the same machine, then that will never be as fast as one NDB node running native on that same machine.
If you are using the virtualization environment as a way to partition customers, consider that MySQL already does this well. A web hosting shop can have a singkle MySQL server and give each customers a database or databases and a user / pass to access only their own stuff.
What is high-end about the database is you are trying to get high-end performance. Having all the data in RAM is the "close to the hardware" part of getting that performance. Having multiple computers is the parallelism part of that performance. Having virutualization sticks a software layer between MySQL and the RAM, and if you have other VMs running on the hardware, then any performance numbers will be dependent on what the other VMs are doing. Of course, if you have all the NDB nodes running as VMs on the same machine, then that will never be as fast as one NDB node running native on that same machine.
If you are using the virtualization environment as a way to partition customers, consider that MySQL already does this well. A web hosting shop can have a singkle MySQL server and give each customers a database or databases and a user / pass to access only their own stuff.
well, if you needed 32GiB ram, sure. But most things don't need 32GiB of ram, and it just doesn't make any sense to host servers with less ram than that.
So yeah, a dedicated server would be better, but probably not all that much better than a properly configured virtual, and the virtual is a whole lot cheaper.
So yeah, a dedicated server would be better, but probably not all that much better than a properly configured virtual, and the virtual is a whole lot cheaper.
The database that I am trying to work into a master-slave replication environment currently has a 500 GB InnoDB file. ( I think since some cleanup was done, the database no longer has that much data in it, but the InnoDB file doesn't shrink. We could dump / undump it to get a clean InnoDB, but it takes 24 hours to dump it and about that to read it in, and we can't be down for multiple days.)
If the virtual nodes are on the same hardware, you also loose the parallelization speedup and the reliability from redundancy, but you have them spread accross physical machines I presume ?
All that said, your setup is probably good enough to learn on and do some testing. I wish I had some real-world load to throw at it.
I think you are right that you could sell NDB MySQL as a service; there is a demand for it, particularly the reliability and allways-up aspects, I think. I think that for people who want more performance, you would do better selling a set of pre-configured machines to deliver to them, and some consulting time, perhaps in partnership with someone who specializes in MySQL performance tuning.
I think that could be pretty lucretive. It would all be a matter of getting the setup down pat and then finding those customers (the hard part).
If the virtual nodes are on the same hardware, you also loose the parallelization speedup and the reliability from redundancy, but you have them spread accross physical machines I presume ?
All that said, your setup is probably good enough to learn on and do some testing. I wish I had some real-world load to throw at it.
I think you are right that you could sell NDB MySQL as a service; there is a demand for it, particularly the reliability and allways-up aspects, I think. I think that for people who want more performance, you would do better selling a set of pre-configured machines to deliver to them, and some consulting time, perhaps in partnership with someone who specializes in MySQL performance tuning.
I think that could be pretty lucretive. It would all be a matter of getting the setup down pat and then finding those customers (the hard part).
A few years back, when MySQL 5.0 was in late beta, I set up a MySQL ndb cluster to handle Dspam for a fairly large mailserver I was in charge of. It seemed to work well, and was really fast. I thought this would overrun the other database solutions with the ease of a man outrunning a small child.
In the intervening years, I've been less involved with databases. I've only needed to touch them for accounting and other performance insensitive applications.
Anyhow, I find the problem interesting. As far as I can tell, NDB is awesome, but I have nothing but synthetic benchmarks. Real load is different.
anyone with real load willing to test that with me? I can donate 3 CPUs and 9-17GiB ram to the project. I would implement on my Xen-based VPS platform[1]. (I'd have the servers on separate power; I can even put them on separate uplinks)
my motivation: well, if I'm wrong, I want to know so I quit giving people shit. If I'm right, and NDB is the answer to MySQL performance problems, I'd like the publicity of saying so.
Also, if the MySQL NDB cluster is as good as I think it is, I'd like to sell it as a service. It's a natural choice, as I'm already setup to host as much ram as possible at the lowest cost possible.
I'm looking for someone who currently has a database that is on a MySQL server with around 4GiB ram, that can't respond to queries in an acceptable timeframe, or perhaps someone who has multiple MySQL servers. If the queries are write-biased, even better (I suspect that regular mysql will do about as well as mysql cluster when it comes to reads, due to filesystem caches, until you start adding more nodes.)
If the experiment is successful, I can continue hosting you for free. The MySQL cluster hosting service will require credibility.
[1]http://prgmr.com/xen/