I totally agree there are a lot of big apps running on Sparc... but I think that it would be much cheaper to buy new x86 hardware for a all products/systems going forward (i.e., those written for MongoDB). Sun (errr... Oracle) also sells x86 Solaris servers (a platform MongoDB supports).
Indeed you can distribute data to multiple independent RDBMS, but balancing when new nodes are added is probably a manual process (or a lot of custom code) that is likely to require downtime. To avoid downtime, your application would need to write to both chunks while it is balancing/migrating (and then delete the old data/chunks once it is migrated to a the RDBMS). Essentially, you would need to write what is already in MongoDB.
You would also have to write a parallel query engine.
I too am a fan of simple designs, but I think rolling your own sharding on top of a RDBMS would likely be a massive chunk of time.
There are really expensive commercial products working on horizontally scaling RDBMS... but personally, I prefer open source and document oriented databases :-)
With MongoDB, you are able to shard your system. This means you can grow your databases horizontally. This is not something you can easily (or cheaply) do in the world of RDBMS. You will see much better scalability with MongoDB than with something like MySQL.
I don't see anyone nipping at their heels. They have some competition in the key/value space, but there are a few different KV paradigms.
MongoDB is a complete replacement for most RDBMS apps and they are way ahead of the pack.