The density of the DIMMs is where the price difference jumps. Right now 32GB sticks push the price way up. Common current chassis have 24 slots letting you hit 384GB total with no big surprises on 16GB sticks.
Getting to 1TB would certainly require 32GB sticks and would be the point where multiple machines gets cheaper.
MailChimp is a DIY email-newsletter service based in Atlanta that serves more than 4 million users worldwide. We're self-funded and profitable, and we're growing fast.
Job Description
MailChimp is seeking a database-focused infrastructure engineer to join our team. You’ll build, maintain, and monitor systems that support millions of users across all products and applications at the company. We operate in a high volume environment and are growing rapidly, adding 10000+ new accounts/day with that rate increasing every week. On the infrastructure side we do interesting work to support that scale and growth rate, working closely with developers to build and support our applications.
The engineering teams are small and focused, our benefits are unmatched, and internally we function much like a startup aside from established stability and abundant resources. There are no sales people, no investors, no board, and engineering teams are trusted to make good decisions with resources with minimal oversight. We are competitive nationally on salary and benefits.
Applicants should have strong Linux experience, extremely heavy operational MySQL experience, independent troubleshooting skills, and a love for automation and monitoring. Our core persistence layer consists of hundreds of horizontally sharded and paired Percona 5.5 instances. We are seeking an experienced engineer who would be part of the core infrastructure team with a focus on maintaining and improving this crucial MySQL layer in addition to contributing to the systems that feed off of it (emailgenome.org, Elasticsearch, Postgres, Redis). We take a pragmatic and practical approach to our stacks using proven components and building our own logic and complexity on top of well understood building blocks. This is a large, well built setup with consistent hardware, configs, monitoring, backups, and tuning in place.
Skills and Experience:
- Linux servers
- Experience with modern MySQL at high volume: sharding, replication, backups, monitoring, tuning
- Knowledge of the current MySQL ecosystem and an interest in where things are headed
- Deep understanding of InnoDB performance, recovery, tuning, and backups
- Experience with replication and an understanding of all options and components involved
- Exposure to HA solutions like MMM, haproxy, PXC
- Scripting and coding (python, bash, php)
- Puppet or other configuration management tool experience is a big plus
- Zabbix or other monitoring tools
- Familiarity with other infrastructure pieces of our stacks (nginx, apache, memcache, postgres, redis, elasticsearch) is also a big plus
Please apply here, the applications come directly to me:
MailChimp is a unique place. We have ~3.5mm users, send ~6bn emails/month, and sit at ~80k queries/second hitting our dozens of database shards during a typical day. We are growing rapidly, adding 7000+ new users/day with that rate increasing every week. On the infrastructure side we do some neat stuff to support that scale and growth rate, working closely with developers to build and support our applications.
The engineering teams are still small, our benefits are unmatched, and internally we function much like a startup aside from established stability and abundant resources. There are no sales people, no investors, no board, no phones, no useless meetings, and engineering teams are trusted to make good decisions with resources with minimal oversight.
If interested, use the links above to apply. Your information will come directly to me.
MailChimp is looking for engineers to join our team. This is a full time position in Atlanta that will help build, support, and monitor the infrastructure our company depends on. We handle tremendous volume and support millions of users that love our products.
We are looking for people with independent troubleshooting skills, strong experience with Linux, and a desire to monitor and automate everything.
Skills & Requirements
Linux experience, especially at higher server counts
Scripting and coding (bash, python, ruby)
Familiarity with pieces of our primary stack (nginx, apache, php, memcache, mysql)
Experience building high volume systems is a big plus
Strong experience with mysql is a huge plus (sharding, replication, HA)
About MailChimp
MailChimp is a self-funded and profitable Atlanta-based company that is growing fast. We offer competitive salaries, exceptional benefits and perks, phone plan coverage, coffee, snacks, top tier equipment, and an environment that empowers engineers to have a big impact. We work in small teams, there are no project managers, no product managers, and engineers are trusted to work autonomously and make good decisions.
Atlanta, GA - Full time, local, relocation possible
MailChimp is looking for infrastructure engineers to join our team. This is a full time position in Atlanta that will help build, support, and monitor the infrastructure our company depends on. We handle tremendous volume and support millions of users that love our products.
We are looking for people with independent troubleshooting skills, strong experience with Linux, and a desire to monitor and automate everything.
- Linux experience, especially at higher server counts
- Scripting and coding
- Familiarity with pieces of our primary stack (nginx, apache, php, memcache, mysql)
- Experience building high volume systems is a big plus
- Strong experience with mysql is a huge plus (sharding, replication, HA)
MailChimp is a self-funded and profitable Atlanta-based company that is growing fast. We offer competitive salaries, exceptional benefits and perks, phone plan coverage, coffee, snacks, top tier equipment, and an environment that empowers engineers to have a big impact. We work in small teams, there are no project managers, no product managers, and engineers are trusted to work autonomously and make good decisions.
Also, in addition to the above, I am looking for somebody with tremendous networking and colo experience.
You can email me directly at infrastructurejob@mailchimp.com
I am looking for Infrastructure Engineers to join the team. We support hundreds of servers, millions of customers, and send billions of emails every month with a small team that prefers automation over manpower.
MailChimp offers extremely competitive pay, unmatched benefits, and a culture that empowers engineers to work autonomously with large budgets and significant resources. We use top of the line equipment to support impressive volume in an international, 24/7 environment.
I am looking for two types right now. Generalists or somebody that can hit both of these are especially welcome:
- Devops, server guys that can write code and contribute to our automation tools, expertise with databases is a large plus given those are our largest machine type.
- Network Engineers, people who absolutely understand and love working with high end networking gear, setting up colocation environments, etc.
We will cover relocation expenses completely for the right candidate and can offer compensation appropriate for any level of experience.
If any interest email infrastructurejob@mailchimp.com and it will come directly to me.
I think the focus on in-memory only is what makes Redis great and I hope they never change.
It lives on that requirement and is incredibly stable and fast. I don't want another half-baked data store that technically "works" on disk but only as long as your volume is completely trivial. There are plenty of those if capacity is a dominating concern.
Softlayer offers this as well. You can spec out a box however you want up to pretty exotic configurations, get billed monthly, and can SSH into them in 3-4 hours.
It is expensive, but not as expensive as something like heroku or aws for equivalent cores/RAM, and you get real hard drives.
We are using Softlayer and we found that on our server we were not bottlenecked by memory, CPU or disk I/O. However, we were bottlenecked by our bandwidth which sat around 5 MB/s. Perhaps we could upgrade to a higher-throughput network card or something (I don't maintain our servers), but we ended up just offloading static files to a CDN instead.
I have a 1.5 TB Postgres database, sharded by schema, that runs wonderfully on a single box (12 core, 36GB RAM, raid 10 of 15k SAS drives). Why couldn't you shard with Postgres?
Good point, sounds like I intend to keep it that way for this particular database in my comment.
It's setup for multi box (each schema is mapped to a hostname in code) but I simply haven't had a reason to move to more boxes yet. The schema feature is a nice, convenient way to pre-shard like this so that growing to more boxes doesn't require rehashing for a very long time if ever (depending on how much sharding you do up front). You just move schemas/shards as needed using the standard dump and restore tools and update the schema->hostname mapping in the code.
If I need to go cross-shard then I am doing it in code. If you knew both shards were on the same box you could do cross-shard joins if you used schemas like this but you would need some potentially tricky logic that determines if it is working with all shards on the same machine.
Thankfully most of the joins happen within a shard (hashing and sharding on something like a user_id) with the exception being various analysis and aggregation queries.
Using PostgreSQL's schemas is admittedly not too different from just using many DBs in MySQL or something else but in practice I've found that extra layer of organization helps keep things neater. I can backup, move, or delete a specific schema/shard or I can backup, move, etc all shards on a machine by operating on the containing database.
Getting to 1TB would certainly require 32GB sticks and would be the point where multiple machines gets cheaper.