I agree with you; a lot of the reason it took us a while to get out of EC2 was my lack of comfort. If I had pushed that earlier many things would have been easier. There are so many great things about EC2, but it pushes the need to think about scaling earlier in many cases, which means that you're thinking about scaling when you still need to be concentrating 100% on product/market fit. We were very fortunate that the first product we built fit the market well, but if we had been wrong the delay could have cost us.
I definitely could have been more clear on that. Cassandra has so many great properties, and when we made the decision to use Postgres for the large dataset under question was shortly after 0.7 was released, and it took a while to get more stable.
The Heroku guys are doing some seriously awesome things with Postgres. I can't wait to see where they go with it; the whole community will benefit, I'm sure.
We make heavy use of queue and internal messaging systems. This discussion was solely about the data storage layers, which always causes us more headaches than the messaging/queuing systems.
Sorry about that -- I couldn't speak as much about our Hbase experience, but I can speak to the toll its taken on our ops team so far, which is very high, with lots of crashes.
You are very right; I should have tried to have us move to physical hardware before we did. Definitely one of the things that I would have done differently, in hindsight.
The larger EC2 instances (especially for always-on systems like primary databases) do get quite a bit cheaper with the 1-year reserved instance reservations, so if you are on EC2 be sure to get those as soon as you're at a somewhat stable point.
Thanks for the kind words, everyone. When I was writing the talk I was worried it'd be too tied to our/my experience, but at least people found it a little entertaining. The video should be posted soon; the Q&A was interesting and brought up some pieces that I glossed over during writing.
According to the docs, this is a blocking operation -- so it's inplace, but I wouldn't consider it to be online, as I would PostgreSQL's vacuum or Cassandra's major compactions:
http://www.mongodb.org/display/DOCS/compact+Command
We made the switch before 1.8 was released, and made the decision during the 1.6 cycle, give or take. Auto-sharding never worked reliably in our tests. I think schmichael does a good job explaining the new features that would have helped -- spares indexes, in particular -- but the core issues still remain (lock contention, no non-blocking online compaction).
getLastError (or safe=true) does not force the write to the disk. Checking getLastError determines whether the server accepted the write, or whether there were failures (uniqueness checks, for instance, or whether any rows were updated).
Do you mean connections to the queue server, or outgoing connections? In either case, it would affect it only slightly (I imagine) if the connections were persistent, as they are in this case. If you're regularly opening and closing connections, the SSL handshake is far more expensive.
I'm a little confused by this; how do SQL databases not support more atomic operations that MongoDB, seeing as you have full control of the transactions themselves? Incrementing, adding to a list (generally done with inserting a row in another table), etc., are all standard operations.
It's a Federal program, but the states have to implement it. Only Delaware, Maine, Maryland, New Jersey, New York, Oregon and Pennsylvania do. In short, if you're on unemployment and you qualify (they have some calculations that determine whether they think you're likely to exhaust primary unemployment anyways) then you can retain your unemployment benefit while starting a company, instead of looking for jobs.
One of my co-founders and I were on this after our last employer went under, and it helped a lot, keeping us focused on building the business instead of taking consulting work to pay the rent.
A good question. Shrinking TCP buffer sizes can have a negative performance impact when sending large amounts of data; our use case was keeping track of a large number of mostly silent connections, and so we benefit from the smaller memory footprint.
It's a spike as in a spike solution -- an end-to-end prototype to see if a chosen method of solving the problem is feasible, but without writing tests or handling all error cases or the like.
MongoDB has a single, global read/write lock for the entire database. So when a write is actually processing, nothing else can happen. In a moderately write-heavy environment, a single slow write can result in pretty bad performance in my experience. Reads can happen at the same time, although there have been problems with not yielding the lock on scans, although there are changes in 1.6 to address that.
Here's the initial note from the forum. I copied it before and can't reload, so I don't know the user that posted it.
"During a shared library update distributed to our hosts, a number of the hosts incorrectly have marked Linodes as being shut down. To recover from this we may be issuing host reboots to upgrade their software to our latest stack, and then bringing the Linodes to their last state. We're working on this now and expect to have additional updates shortly. We'll also be notifying those affected via our support ticket system. Please stand by."