Moving 6 Billion Messages Without Being Noticed(dt.deviantart.com)
dt.deviantart.com
Moving 6 Billion Messages Without Being Noticed
http://dt.deviantart.com/art/Moving-6-Billion-Messages-Without-Being-Noticed-285571516
1 comments
They needed the Percona toolkit (previously Maatkit). http://www.percona.com/doc/percona-toolkit/2.0/pt-table-sync...
Disclaimer: I wrote that article.
Percona toolkit has some useful stuff. Thanks for the link.
In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.
Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.
Percona toolkit has some useful stuff. Thanks for the link.
In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.
Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.
The toolkit documentation reports limitations.
What they really needed is ChronicDB http://chronicdb.com
What they really needed is ChronicDB http://chronicdb.com
If you are going to flog ChronicDB, please at least disclose if you have some sort of relationship with said company. Most of your previous submissions pertain to ChronicDB.
Sorry for not disclosing I work on ChronicDB.
The author of the 6b article kindly provided more information about the time it took for the migration:
http://news.ycombinator.com/item?id=3621561
A couple of limitations of the Percona tool are related to changing data in tables with ON DELETE and ON UPDATE foreign key constraints and possible lock-ups of tables. It is a bi-directional replication tool, so it has to deal with the master-master replication case and as such does not guarantee data consistency.
Still the tool is helpful in many cases and congrats to Percona for developing it. Replication safety is a non-trivial problem in general.
The author of the 6b article kindly provided more information about the time it took for the migration:
http://news.ycombinator.com/item?id=3621561
A couple of limitations of the Percona tool are related to changing data in tables with ON DELETE and ON UPDATE foreign key constraints and possible lock-ups of tables. It is a bi-directional replication tool, so it has to deal with the master-master replication case and as such does not guarantee data consistency.
Still the tool is helpful in many cases and congrats to Percona for developing it. Replication safety is a non-trivial problem in general.
They weren't doing a master-master sync.
ChronicDB is a hosted solution. That isn't even remotely what DeviantArt were looking for.
ChronicDB is not only a hosted solution. It also runs locally.
Not according to your own website.
ChronicDB is a combination of a software engineer and a hosted database: You run your database in our datacenters -- http://chronicdb.com/how_it_works
Your database will be hosted in our datacenters -- http://chronicdb.com/features
ChronicDB is a combination of a software engineer and a hosted database: You run your database in our datacenters -- http://chronicdb.com/how_it_works
Your database will be hosted in our datacenters -- http://chronicdb.com/features