Most of Kubernetes clusters I've dealt with wastes >40% of its provisioned resources to fragmentation. It's a shame this become kinda normalized. Tools like Karpenter helped (compared to old days of CAS), however, bin-packing efficiency depends on so many factors (e.g Nodepool design, Pod churn rate, etc) and usually needs to be tuned to each cluster profile.
I built kube-binpacking-exporter to easily track the most important metrics when improving bin-packing, it's like running eks-node-viewer in a loop and exporting metrics to Prometheus (or any O11Y tool). It's not a generic exporter you don't have to be using Karpenter.
While these bin-packing metrics can be calculated with the combination of `kube-state-metrics`, `kubelet` and `cAdvisor` metrics they fall short because:
1. These metrics are pulled from different sources at different intervals. This causes aggregation to not give an accurate snapshot of the cluster state per scrape. When aggregating over long periods of time (days+) the inaccuracies compound.
2. Queries get extremely complex, and you have to handle many cases ( e.g exclude failed & completed pods, handle init containers, not count pending pods, and will need complex `joins` to group by node labels )
3. Some O11Y tools query language ( looking at you Datadog!!!! ) lacks the flexibility to join & combine metrics from different data sources.
Please some one help [redacted]! I am so far away from him and it break my hear seeing him destroying himself and his business! I love [redacted], it made me a better engineer especially in my Country!
I built kube-binpacking-exporter to easily track the most important metrics when improving bin-packing, it's like running eks-node-viewer in a loop and exporting metrics to Prometheus (or any O11Y tool). It's not a generic exporter you don't have to be using Karpenter.
While these bin-packing metrics can be calculated with the combination of `kube-state-metrics`, `kubelet` and `cAdvisor` metrics they fall short because:
1. These metrics are pulled from different sources at different intervals. This causes aggregation to not give an accurate snapshot of the cluster state per scrape. When aggregating over long periods of time (days+) the inaccuracies compound.
2. Queries get extremely complex, and you have to handle many cases ( e.g exclude failed & completed pods, handle init containers, not count pending pods, and will need complex `joins` to group by node labels )
3. Some O11Y tools query language ( looking at you Datadog!!!! ) lacks the flexibility to join & combine metrics from different data sources.