I'm a Sr Dev here at Datarank in charge of our ES architecture. Maybe I can shed some light on your points.
"I was under the impression that shards are kept uniform in size as ES will try and equally spread data to all shards in an index, so that there aren't imbalances."
This is true by default and it's fine for simple document retrieval but it doesn't scale well if you want to do complex aggregations on arbitrary filters of large datasets. For that you need document clustering. In our case (and probably others), the clustering can't be done uniformally, see http://engineering.datarank.com/2015/06/30/analysis-of-hotsp....
"Writes initially only happen on primary shards, and then propagate to replicas." - also true but you still want this to be as distributed as possible. In some cases we are bulk loading millions of documents. We want the load to be equally distributed as possible while still allowing for clustering as mentioned above. Also, we want to minimize heap usage per node for GC and buffering performance.
"At some point you'll have to redesign your ES architecture." - Perhaps but this system scales to 100s nodes easily and maybe 1000s. We allow our customers to perform very complex aggregates on some pretty large datasets with 50ms response times. All other architectures we tried failed to scale well.
Clustering of comments is vital to our use case for performance. Our data is distributed log-normally. We must be able to scale quickly and easily. The default setup didn't scale well with our volume. All of these factors lead us to this solution.
"I was under the impression that shards are kept uniform in size as ES will try and equally spread data to all shards in an index, so that there aren't imbalances."
This is true by default and it's fine for simple document retrieval but it doesn't scale well if you want to do complex aggregations on arbitrary filters of large datasets. For that you need document clustering. In our case (and probably others), the clustering can't be done uniformally, see http://engineering.datarank.com/2015/06/30/analysis-of-hotsp....
"Writes initially only happen on primary shards, and then propagate to replicas." - also true but you still want this to be as distributed as possible. In some cases we are bulk loading millions of documents. We want the load to be equally distributed as possible while still allowing for clustering as mentioned above. Also, we want to minimize heap usage per node for GC and buffering performance.
"At some point you'll have to redesign your ES architecture." - Perhaps but this system scales to 100s nodes easily and maybe 1000s. We allow our customers to perform very complex aggregates on some pretty large datasets with 50ms response times. All other architectures we tried failed to scale well.
Clustering of comments is vital to our use case for performance. Our data is distributed log-normally. We must be able to scale quickly and easily. The default setup didn't scale well with our volume. All of these factors lead us to this solution.