Half of Organizations Spend More with Kubernetes, Mostly Due to Overprovisioning(infoq.com)
infoq.com
Half of Organizations Spend More with Kubernetes, Mostly Due to Overprovisioning
https://www.infoq.com/news/2024/03/cncf-finops-kubernetes-overspend/
7 comments
Agreed about opex while the business is ramping.
Kubernetes has universally been a productivity killer in the organizations I’ve seen deploy it. It’s one of the bigger threats to the company at my current gig.
Maybe it’s just stuck in the trough of disillusionment this year, but I’d love to see a non-terrible replacement for it take over.
Kubernetes has universally been a productivity killer in the organizations I’ve seen deploy it. It’s one of the bigger threats to the company at my current gig.
Maybe it’s just stuck in the trough of disillusionment this year, but I’d love to see a non-terrible replacement for it take over.
It's been the opposite for us. No more waiting for VMs to get spun up and it actually enforces a consistent layer across infrastructure.
I'd be curious what the pain you've seen is.
I'd be curious what the pain you've seen is.
Likely just a lack of organizational expertise.
Also, if you use some of the tools out there advertised to make K8s easier (operators + Helm) - you actually do have a harder time debugging the thing if you don’t know K8s well due to how many layers of abstraction those tools add.
Also, if you use some of the tools out there advertised to make K8s easier (operators + Helm) - you actually do have a harder time debugging the thing if you don’t know K8s well due to how many layers of abstraction those tools add.
The Hashicorp suite (Nomad as the orchestrator) has the reputation to be simpler and more capable of scaling but I've never tried it yet. Better hurry since the industry has apparently decided to flock around kubernetes.
what would you say are the biggest sources of pain? k8s is definitely a beast, and I'm curious what parts in particular that drag down productivity for you.
It is a complicated beast, extremely flexible. It's a box of Legos you need to assemble yourself. There is space for simpler, opinionated solutions (such as OpenShift) that require making fewer decisions and keeping track of less configuration.
yeah ok that's about what I figured.
imho you can't really raw dog k8s unless you've got a team dedicated to building tools or, to use your analogy, packaging up sets out of the messy box of Legos and writing assembly instructions.
imho you can't really raw dog k8s unless you've got a team dedicated to building tools or, to use your analogy, packaging up sets out of the messy box of Legos and writing assembly instructions.
"Respondents shared that they see team and individual awareness and self-discipline as the best ways to help get the overspend under control (68%)"
Another benefit of running small, cross-functional teams. It's a lot more difficult for bloat to become overwhelming when teams have strong autonomy over their own decisions and stacks. Think this will naturally shift down as larger orgs make switch to small, 8-12 person autonomous teams.
Another benefit of running small, cross-functional teams. It's a lot more difficult for bloat to become overwhelming when teams have strong autonomy over their own decisions and stacks. Think this will naturally shift down as larger orgs make switch to small, 8-12 person autonomous teams.
> Another benefit of running small, cross-functional teams. It's a lot more difficult for bloat to become overwhelming when teams have strong autonomy over their own decisions and stacks
I’ve found the opposite to be true unless autonomy is paired with accountability. Ultimately the people making decisions need to pay the costs for their choices and I’ve seen that be especially bad where someone had the autonomy to spin up a ton of resources which someone else had to pay for. This can be especially bad if that was resume-driven development where they were playing cloud service bingo trying to list experience with things they really couldn’t have justified based on the application’s actual usage.
I’ve found the opposite to be true unless autonomy is paired with accountability. Ultimately the people making decisions need to pay the costs for their choices and I’ve seen that be especially bad where someone had the autonomy to spin up a ton of resources which someone else had to pay for. This can be especially bad if that was resume-driven development where they were playing cloud service bingo trying to list experience with things they really couldn’t have justified based on the application’s actual usage.
autonomy will absolutely lead to bloat if there isn't an accountability mechanism in place.
at my last job as head of eng at a small (15-20 devs), I pushed teams to build their dashboards with business metrics in addition to the usual operational metrics. things like costs, customer frustration signals and support ticket volumes, product metrics, etc would show up there.
integrating these initially is non-trivial, but once the data is flowing, it's easy for teams to pick them up. gives a good sense of ownership, helps teams address these proactively (or build a business case for addressing them), and gives the team+stakeholders a common dataset so everyone has the same complete picture to work from.
at my last job as head of eng at a small (15-20 devs), I pushed teams to build their dashboards with business metrics in addition to the usual operational metrics. things like costs, customer frustration signals and support ticket volumes, product metrics, etc would show up there.
integrating these initially is non-trivial, but once the data is flowing, it's easy for teams to pick them up. gives a good sense of ownership, helps teams address these proactively (or build a business case for addressing them), and gives the team+stakeholders a common dataset so everyone has the same complete picture to work from.
I wouldn't say this is a problem specific to kubernetes.
Most places over-provision their virtual machines also, if not more so due to the less dynamic state of the lower layers.
Most places over-provision their virtual machines also, if not more so due to the less dynamic state of the lower layers.
[deleted]
I would wager that more than half of all Kubernetes deployments are completely unnecessary yak shaves.
I think there is a solid argument for it as a cloud portability layer – I’m literally in the middle of a GKE->EKS migration where that’s paying off as hoped - but I would fascinated to see the numbers for what percentage of places ever do that or the total cost incurred avoiding services which couldn’t be used interchangeably between providers. I think people want ORM-level portability but it’s more like how you can technically have a Java desktop app support Windows, macOS, and Linux.
The portability of your stack is one aspect but there are other more important factors:
1. The tools you want to use work with the other k8s components or things commonly available in k8s environments. Say you want a cloud-portable messaging system. You can deploy Apache Pulsar on k8s and if you use the operator, helm charts (eww) or templates it will integrate out of the box with Prometheus, cert-manager, whatever volume provisioner you are using etc. You can't get that level of cloud-agnostic deployability or integration with tools like Terraform/Pulumi because all the underlying infrastructure resources are different on a per-cloud basis. Having such a powerful generic API that properly describes like 99% of workloads is a massive game changer.
2. The skillset is transferable. If you are comparing k8s to something like ECS, Cloud Run or friends you will end up with a bunch of proprietary knowledge that is hard to transfer. Similarily your hires won't come with that knowledge either but in this day and age they probably have robust k8s skills if they are infra folk or are aware of it if above that part of the stack.
3. Flexibility. Perhaps it's greatest strength and greatest weakness is that the set of primitives chosen to be exposed by k8s (with the exception of things like Ingress that needed to get re-done) are by and large skewed towards flexibility. This means doing all manner of 'very difficult things' are now considerably easier. Conversely however this flexibility means you still need to fully describe more common patterns which simpletons mistake for being "over-complicated" when what they really mean is "less easy". Don't fall into the trap of equating "easy" to "simplicity", k8s once understood is by far one of the simplest but as a result is usually more work to get started. Good example is say Arch Linux vs MacOS, the former is simple the latter is easy.
1. The tools you want to use work with the other k8s components or things commonly available in k8s environments. Say you want a cloud-portable messaging system. You can deploy Apache Pulsar on k8s and if you use the operator, helm charts (eww) or templates it will integrate out of the box with Prometheus, cert-manager, whatever volume provisioner you are using etc. You can't get that level of cloud-agnostic deployability or integration with tools like Terraform/Pulumi because all the underlying infrastructure resources are different on a per-cloud basis. Having such a powerful generic API that properly describes like 99% of workloads is a massive game changer.
2. The skillset is transferable. If you are comparing k8s to something like ECS, Cloud Run or friends you will end up with a bunch of proprietary knowledge that is hard to transfer. Similarily your hires won't come with that knowledge either but in this day and age they probably have robust k8s skills if they are infra folk or are aware of it if above that part of the stack.
3. Flexibility. Perhaps it's greatest strength and greatest weakness is that the set of primitives chosen to be exposed by k8s (with the exception of things like Ingress that needed to get re-done) are by and large skewed towards flexibility. This means doing all manner of 'very difficult things' are now considerably easier. Conversely however this flexibility means you still need to fully describe more common patterns which simpletons mistake for being "over-complicated" when what they really mean is "less easy". Don't fall into the trap of equating "easy" to "simplicity", k8s once understood is by far one of the simplest but as a result is usually more work to get started. Good example is say Arch Linux vs MacOS, the former is simple the latter is easy.
I’d class your first two points as portability but otherwise agree. There are solid business cases for picking Pulsar vs. e.g. AWS SQS/SNS and a lot of that is going to come down to whether you value portability more than the increased operational commitments. I wouldn’t fault anyone for picking either side of that as long as it was a considered call.
I guess if you consider k8s to be it's own ecosystem and then frame the first two points as just a consequence of the ecosystem being inherently portable then I guess so?
IMO though compatibility and skill transferability (which speaks mostly to adoption rate) are separate things.
IMO though compatibility and skill transferability (which speaks mostly to adoption rate) are separate things.
Definitely separate things - I just view them as part of the larger question of where you fall on the spectrum of how much you are outsourcing to your cloud provider.
Another way of approaching it: would the cost of porting, say, an app which uses Lambda/SQS/SNS to GCP/cloud functions/PubSub be greater or lower than the cost of maintaining your own FaaS & queue/notification service on top of k8s? The answer isn’t fixed and will depend on things like your usage and staffing, and how complex your applications are, so I’d be most curious about how commonly people pick one versus the other and regret it due to some factor they didn’t anticipate.
Another way of approaching it: would the cost of porting, say, an app which uses Lambda/SQS/SNS to GCP/cloud functions/PubSub be greater or lower than the cost of maintaining your own FaaS & queue/notification service on top of k8s? The answer isn’t fixed and will depend on things like your usage and staffing, and how complex your applications are, so I’d be most curious about how commonly people pick one versus the other and regret it due to some factor they didn’t anticipate.
I suspect a big chunk of this is simply visibility: if you deploy an application using your cloud provider’s native features, you get an itemized bill telling you where your money is going. That’s not perfect (you still need to tag resources) but it’s mature and familiar. If you run everything in Kubernetes - which, to head off a popular tangent, is not just following a fad but a trade off - you have to build and operate the equivalent infrastructure to have the same level of visibility, but since the entire Kubernetes cluster is overhead it’s very easy to get stuck in the IT shared services pitfall where there’s a constant pressure to direct limited resources towards application-visible work.
The survey’s reported breakdown feels right – you need solid leadership to resist the urge to skimp on indirect infrastructure costs and I’d bet most of the 80% know this is a problem but have been too busy to spend time on it:
> just under a fifth (19%) had access to accurate information, while 40% said they estimated their Kubernetes costs. There was no monitoring in place for 38%.
The survey’s reported breakdown feels right – you need solid leadership to resist the urge to skimp on indirect infrastructure costs and I’d bet most of the 80% know this is a problem but have been too busy to spend time on it:
> just under a fifth (19%) had access to accurate information, while 40% said they estimated their Kubernetes costs. There was no monitoring in place for 38%.
kube-prometheus-stack will setup Prometheus and grafana and once you get the hang of it makes sizing resources a bit better
Sure, but that’s something new to install, configure, and maintain. My point is simply that a lot of places struggle to fund those kind of internal services because spending time on them doesn’t directly produce application-level benefits.
I wonder if there are still companies out there running tomcat 10 clusters on their dedicated servers and furiously laughing at the depressing state of the software engineering world.
"Can't be blamed for using IBM" companies are very likely to be using Kubernetes these days, though.
Even in pure java shops, you‘d definitely get laughed out of most rooms for proposing a tomcat cluster. Majority of java devs nowadays probably never even experienced working with it directly. There is little knowhow of it left.
Everything must be dockerized and dockerized stuff must run in the cloud.
Everything must be dockerized and dockerized stuff must run in the cloud.
For most folks here on HN, you should not care about your cloud opex cost as long as it is time benefiting. The most costly thing to waste is time.
Can you cut your cloud bill by 35% with optimal provisioning? Most likely. Can you do this cut with a greater return given all the inputs, time, wages, opportunity cost, AND what level of confidence in completion to that exact or less deadline?
The only times I've seen the numbers work out has been, growth is flattening, +$3M/yr vendor spend or $45k/month specific service spend.