Pro tip: try to figure out the delicious accounts of your friends, coworkers, and generally interesting online people. Add these people to your delicious network and then subscribe to your own network with your feed reader of choice. It's an easy and passive way to harvest lots of great links.
Sage Weil, the creator of Ceph, recently gave a talk at linux conf au. The slides [1] mostly stand on their own. The first two slides address your first question. The remainder of the slides are a deeper dive into the technical aspects of the system than TFA gives.
I am not using Ceph but my company makes a system built on an extremely similar architecture and I can vouch for the scalability (modulo implementation stupidities, natch). That being said, our system is difficult to administer due to all the moving parts and its distributed nature - I don't know if Ceph shares this drawback.
I think you're probably right about ensemble setup overhead not mattering very much if you are creating a cluster in your own datacenter for your own apps. In my case, I need to bootstrap an ensemble in someone else's datacenter without a priori knowledge of ip addresses.
I don't mean to be totally negative. I really respect what the zk team has done and it's a fantastic tool that is often and unfortunately overshadowed by Hadoop. I think a small, reliable, coordination cluster providing atomic distributed primitives is an excellent distributed system architecture. Obviously Google and Yahoo have had success with it. I just recently learned that Ceph takes a similar approach.
I work on an appliance system that is really a small (<16 node) distributed system. Things like peer discovery, service lookup, leader election, distributed locks, transaction coordinators, pub-sub queues, and a reliable store for small configuration-like data are useful to our system and, I assert, most distributed systems. Zookeeper can provide all of these things.
In my view, Zookeeper's biggest shortcoming is that it doesn't support dynamic membership. That is to say that the Zookeeper nodes themselves are not self-organizing, you have to specify the ip addresses of all Zookeeper cluster members. Issue 107 [1] tracks this deficiency.