[Disclaimer: I work on Quantum ActiveScale, an on-prem S3 system that fits this list]
Yup this is why these vendors exist. You’re definitely not alone, cloud repatriation is a ‘thing’.
These vendors have replaced that Ceph or Minio expert others in this thread said you’d need to budget for, by software. The system detects dead/degrading disks and automatically evicts them and rebuilds that chunk of the erasure code on another disk. Every few months you go in and hot swap the bad disks = the ones with a blinking led. Also Prometheus metrics, alerts in case of issues,... You don’t need a storage admin babysitting this.
1 rack of 4u90 enclosures with 18TB disks is 15PB RAW so with erasure coding overhead about the 10PB usable you need today.
I’m obviously biased on which vendor to pick. Do your due diligence, f.i. on how the system does capacity expansions.
Good discussion of an (indeed) epic article. Reading the other comments here, I would like to stress the complexity of allowing concurrent updates of the same object on distributed machines.
And I totally share the hopes for the current eventual consistency monopoly to perish. Some use cases should use strong consistency. You see people build stuff like this https://github.com/Netflix/s3mper to "fix" eventual consistency...awkward
Yup this is why these vendors exist. You’re definitely not alone, cloud repatriation is a ‘thing’.
These vendors have replaced that Ceph or Minio expert others in this thread said you’d need to budget for, by software. The system detects dead/degrading disks and automatically evicts them and rebuilds that chunk of the erasure code on another disk. Every few months you go in and hot swap the bad disks = the ones with a blinking led. Also Prometheus metrics, alerts in case of issues,... You don’t need a storage admin babysitting this.
1 rack of 4u90 enclosures with 18TB disks is 15PB RAW so with erasure coding overhead about the 10PB usable you need today.
I’m obviously biased on which vendor to pick. Do your due diligence, f.i. on how the system does capacity expansions.