The density of the DIMMs is where the price difference jumps. Right now 32GB sticks push the price way up. Common current chassis have 24 slots letting you hit 384GB total with no big surprises on 16GB sticks.
Getting to 1TB would certainly require 32GB sticks and would be the point where multiple machines gets cheaper.
MailChimp is a DIY email-newsletter service based in Atlanta that serves more than 4 million users worldwide. We're self-funded and profitable, and we're growing fast.
Job Description
MailChimp is seeking a database-focused infrastructure engineer to join our team. You’ll build, maintain, and monitor systems that support millions of users across all products and applications at the company. We operate in a high volume environment and are growing rapidly, adding 10000+ new accounts/day with that rate increasing every week. On the infrastructure side we do interesting work to support that scale and growth rate, working closely with developers to build and support our applications.
The engineering teams are small and focused, our benefits are unmatched, and internally we function much like a startup aside from established stability and abundant resources. There are no sales people, no investors, no board, and engineering teams are trusted to make good decisions with resources with minimal oversight. We are competitive nationally on salary and benefits.
Applicants should have strong Linux experience, extremely heavy operational MySQL experience, independent troubleshooting skills, and a love for automation and monitoring. Our core persistence layer consists of hundreds of horizontally sharded and paired Percona 5.5 instances. We are seeking an experienced engineer who would be part of the core infrastructure team with a focus on maintaining and improving this crucial MySQL layer in addition to contributing to the systems that feed off of it (emailgenome.org, Elasticsearch, Postgres, Redis). We take a pragmatic and practical approach to our stacks using proven components and building our own logic and complexity on top of well understood building blocks. This is a large, well built setup with consistent hardware, configs, monitoring, backups, and tuning in place.
Skills and Experience:
- Linux servers
- Experience with modern MySQL at high volume: sharding, replication, backups, monitoring, tuning
- Knowledge of the current MySQL ecosystem and an interest in where things are headed
- Deep understanding of InnoDB performance, recovery, tuning, and backups
- Experience with replication and an understanding of all options and components involved
- Exposure to HA solutions like MMM, haproxy, PXC
- Scripting and coding (python, bash, php)
- Puppet or other configuration management tool experience is a big plus
- Zabbix or other monitoring tools
- Familiarity with other infrastructure pieces of our stacks (nginx, apache, memcache, postgres, redis, elasticsearch) is also a big plus
Please apply here, the applications come directly to me:
MailChimp is a unique place. We have ~3.5mm users, send ~6bn emails/month, and sit at ~80k queries/second hitting our dozens of database shards during a typical day. We are growing rapidly, adding 7000+ new users/day with that rate increasing every week. On the infrastructure side we do some neat stuff to support that scale and growth rate, working closely with developers to build and support our applications.
The engineering teams are still small, our benefits are unmatched, and internally we function much like a startup aside from established stability and abundant resources. There are no sales people, no investors, no board, no phones, no useless meetings, and engineering teams are trusted to make good decisions with resources with minimal oversight.
If interested, use the links above to apply. Your information will come directly to me.
MailChimp is looking for engineers to join our team. This is a full time position in Atlanta that will help build, support, and monitor the infrastructure our company depends on. We handle tremendous volume and support millions of users that love our products.
We are looking for people with independent troubleshooting skills, strong experience with Linux, and a desire to monitor and automate everything.
Skills & Requirements
Linux experience, especially at higher server counts
Scripting and coding (bash, python, ruby)
Familiarity with pieces of our primary stack (nginx, apache, php, memcache, mysql)
Experience building high volume systems is a big plus
Strong experience with mysql is a huge plus (sharding, replication, HA)
About MailChimp
MailChimp is a self-funded and profitable Atlanta-based company that is growing fast. We offer competitive salaries, exceptional benefits and perks, phone plan coverage, coffee, snacks, top tier equipment, and an environment that empowers engineers to have a big impact. We work in small teams, there are no project managers, no product managers, and engineers are trusted to work autonomously and make good decisions.
Atlanta, GA - Full time, local, relocation possible
MailChimp is looking for infrastructure engineers to join our team. This is a full time position in Atlanta that will help build, support, and monitor the infrastructure our company depends on. We handle tremendous volume and support millions of users that love our products.
We are looking for people with independent troubleshooting skills, strong experience with Linux, and a desire to monitor and automate everything.
- Linux experience, especially at higher server counts
- Scripting and coding
- Familiarity with pieces of our primary stack (nginx, apache, php, memcache, mysql)
- Experience building high volume systems is a big plus
- Strong experience with mysql is a huge plus (sharding, replication, HA)
MailChimp is a self-funded and profitable Atlanta-based company that is growing fast. We offer competitive salaries, exceptional benefits and perks, phone plan coverage, coffee, snacks, top tier equipment, and an environment that empowers engineers to have a big impact. We work in small teams, there are no project managers, no product managers, and engineers are trusted to work autonomously and make good decisions.
Also, in addition to the above, I am looking for somebody with tremendous networking and colo experience.
I am looking for Infrastructure Engineers to join the team. We support hundreds of servers, millions of customers, and send billions of emails every month with a small team that prefers automation over manpower.
MailChimp offers extremely competitive pay, unmatched benefits, and a culture that empowers engineers to work autonomously with large budgets and significant resources. We use top of the line equipment to support impressive volume in an international, 24/7 environment.
I am looking for two types right now. Generalists or somebody that can hit both of these are especially welcome:
- Devops, server guys that can write code and contribute to our automation tools, expertise with databases is a large plus given those are our largest machine type.
- Network Engineers, people who absolutely understand and love working with high end networking gear, setting up colocation environments, etc.
We will cover relocation expenses completely for the right candidate and can offer compensation appropriate for any level of experience.
If any interest email [email protected] and it will come directly to me.
I think the focus on in-memory only is what makes Redis great and I hope they never change.
It lives on that requirement and is incredibly stable and fast. I don't want another half-baked data store that technically "works" on disk but only as long as your volume is completely trivial. There are plenty of those if capacity is a dominating concern.
Softlayer offers this as well. You can spec out a box however you want up to pretty exotic configurations, get billed monthly, and can SSH into them in 3-4 hours.
It is expensive, but not as expensive as something like heroku or aws for equivalent cores/RAM, and you get real hard drives.
If I need to go cross-shard then I am doing it in code. If you knew both shards were on the same box you could do cross-shard joins if you used schemas like this but you would need some potentially tricky logic that determines if it is working with all shards on the same machine.
Thankfully most of the joins happen within a shard (hashing and sharding on something like a user_id) with the exception being various analysis and aggregation queries.
Using PostgreSQL's schemas is admittedly not too different from just using many DBs in MySQL or something else but in practice I've found that extra layer of organization helps keep things neater. I can backup, move, or delete a specific schema/shard or I can backup, move, etc all shards on a machine by operating on the containing database.
Good point, sounds like I intend to keep it that way for this particular database in my comment.
It's setup for multi box (each schema is mapped to a hostname in code) but I simply haven't had a reason to move to more boxes yet. The schema feature is a nice, convenient way to pre-shard like this so that growing to more boxes doesn't require rehashing for a very long time if ever (depending on how much sharding you do up front). You just move schemas/shards as needed using the standard dump and restore tools and update the schema->hostname mapping in the code.
I have a 1.5 TB Postgres database, sharded by schema, that runs wonderfully on a single box (12 core, 36GB RAM, raid 10 of 15k SAS drives). Why couldn't you shard with Postgres?
The tough thing about comparing these two is that it is hard to find a benchmark (I don't know of one) that works similarly with them both. Sysbench is somewhat of a standard for testing MySQL configs but Postgres does not handle it well.
Though I tend to prefer Postgres don't get me wrong - MySQL has some advantages.
For raw pkey lookups, especially range selects against pkeys with lower amounts of concurrency InnoDB's speed is unmatched. It should be given that it is completely laid out on disk specifically for that to the detriment of other features.
MySQL's replication is extremely sturdy. On paper the older method (statement-based) sounds incredibly fragile but in practice it works extremely well and has been proven on countless projects. mmm makes it almost too easy to setup replication and failover.
I feel that MySQL stalled out for several years in the 5.0-5.1 period where poorly engineered features were bolted on to create a product with too many pathological cases that destroyed performance to keep track of. All these features were available but experience taught you to avoid most joins, avoid most subselects, avoid most usages of views, etc.
That said, v5.5 has a lot of nice improvements and can actually scale up to more than 8 cores (I've gotten linear improvement up to about 32 cores and that is what Oracle puts in their white papers as well). Percona and Facebook are releasing nice patches and branches, and forks like Drizzle are reaching GA and doing good things as well. So I think it is headed in the right direction again.
Compared to MyISAM? Sure for certain aggregate functions, but it's too easy to point out reasons for avoiding MyISAM: fragmentation, no transactions, table locks on writes, complete table rewrites to change columns or indexes, only scales up to about 8 cores.
Compared to InnoDB? You are incorrect, Postgres is faster, even for a raw count(*), even when comparing against the InnoDB plugin and not the ancient InnoDB builtin.
I have access to tuned TB+ DBs of both types and am happy to disprove any specific examples you can provide.
Postgres has bitmap index support baked in as well and it works really well. I'd be awfully interested in knowing where MySQL beats it by 50x at anything.
I am not a big fan of Hadoop. It is a headache to configure and optimized for installs with node counts only a few companies could make use of. I really wish there were more options as I believe Hadoop is overkill for most of the people using it.
For quick and dirty map reduce on a smaller node count I've started to really like Disco (discoproject.org). You just pull down the backend with your package manager, push your files into ddfs, write a python script, and run it.
Agreed, hard drives fail constantly. I've had as many as 3 dead at a time while waiting for hot spares to rebuild in arrays as small as 24 drives. Dual drive failures have happened at least twice that I can remember off the top of my head in the last 3-4 years.
Usually running raid10 for performance reasons but when I don't need fast writes I use raid6 instead of raid5 for anything beyond 8 drives.
There are some nice middle options out there. I'll use Softlayer as an example as I have provisioned a lot of machinery over there.
I can order machines online and SSH in 3-4 hours later. Even exotic stuff they turn around just as fast - we saw that speed on a quad octocore box with a raid 10 of Intel SSDs.
That's real metal too, with real IO (most of my work is IO bound so VMs and the cloud are not options). You get to pick the exact CPUs, disks, etc and they slot them in solid Super Micro boards and use good Adaptec disk controllers. You pay monthly and can spin down the box at any time (though must pay full months, no per-minute pricing like AWS).
That is on the dedicated hardware side, you can also spin up compute instances and those can be cloned and fired up in bulk. But, they also have the IO problems that all other VMs have.
In any case, just wanted to mention they are a decent middle ground. Not as automated and polished as Amazon on the VM side but you can spin up mixtures of metal and VMs to get combinations that make sense - pushing compute or RAM-only stuff to VMs and keeping DBs and persistence layers on real metal. They have a few different datacenters too so you can spread gear around physical locations.
Getting to 1TB would certainly require 32GB sticks and would be the point where multiple machines gets cheaper.