Indeed if there were “official” container images out there, I might have instead run the server on Google Cloud Run or AWS AppRunner, without having to take care of the Linux underneath. Or an Amazon ECS task. I don’t have a Kubernetes cluster, but I will at some point make a version of this blog to run it on K8s :-)
> Telemetry does not address this, though. Shoving it into a container and assigning it a simple “restart if down” rule does.
A Systemd unit as shown in [1] does it too without using containers and with fewer moving parts of using containers. I use containers every day at $work. I have been using containers since before Docker was a thing. In this case, it's entirely overkill: Systemd units use the important things like cgroups already.
For the upgrade: depends. You do need a container image regardless, and I have not seen official ones. Upgrading servers in Minecraft requires upgrading clients to match, and my kids prefer to play, more than upgrade. (Unless a biome is released. Then it must be immediately available to them.) But then again, I just need to download the binary with a cURL call. And if the configurations change, Docker won't help me there one bit anyhow.
To be fair, the setup of the article works with most modern observability solutions, in same cases just by replacing the endpoint and authentication token. Turning telemetry processing into a sort of utility is one of the great things that OpenTelemetry did. Now, among vendors, we compete on delivering insights on the telemetry, as opposed to just collecting it. If you are interested, I wrote about it a while back [1].
About excessive telemetry, that depends on what you want to achieve. Using facilities in the OpenTelemetry Collector like [2], you can easily drop all telemetry you have no use for. At the cost of tooting my own horn, we actually provide super easy ways of doing the same dropping at no charge whatsoever to the end user in Dash0 [3].
I actually had to splurge got 2 VCPUs on Digital Ocean to avoid "skipping ticks" and it does sound pretty nuts to me. We play max 3 players. I would expect the server with such a load to be able to run on a slightly tuned up toaster.
> As you said "There is so much fun that could be had in setting up an actually resilient system instead.", maybe the author has more fun setting up alerts and metrics instead of a resilient system like you do?
Adding the backup for the world files, already having Systemd bringing back a crashing server, makes the setup rather resilient. Sure, there's infinite more things that can go wrong, but with swiftly decreasing likelihood.
> The truth is that in most real-world scenarios getting alerts, metrics is much more important than building a fully resilient system (Expensive, maybe overengieering for early stage etc.).
This, very much this.
> However, the author is not really procrastinating—he gets paid for this. As the first sentence in the blog post says "One of the secret pleasures of life is to be paid for things you would do for free.", which I can very much understand as I often work or play with things I could use at work in my free time.
> How are metrics helpful? There is so much fun that could be had in setting up an actually resilient system instead.
Metrics are the means to an end of alerting. And with alerting, I mean getting pinged on my phone when something important breaks. Like, you know, the server going down.
> Why worry over metrics and alerts when you could orchestrate an infrastructure that grants you the superpower of being able to spin up a server with a copy of the world on a whim instead (or even a system that auto-starts one whenever there is demand)?
As somebody who has run cloud and enterprise software for almost two decades now, I can be that needs monitoring too. The more moving parts there are, the more things go wrong. The more things go wrong, and the more you care they get fixed, the more monitoring you need :-)
One of the stretch goals for me writing this article was indeed to show between the lines how Prometheus Exporters, the OpenTelemetry Collector and Systemd can all work together. That is a very reusable skill on monitoring workflows running outside containers on Linux VMs or hosts.
For this, I have the impression that https://github.com/dirien/minectl might be very close to what you are thinking. I did not try it, but took the Minecraft Exporter from it and used in the setup.
I am indeed an employee of Dash0. The setup for telemetry collector will work with anything that accepts OTLP, and with minor adjustments, the data can be sent elsewhere too in other formats, as the OpenTelemetry Collector is very flexible in that regard.
Alerting is specific to Dash0. I know of no other monitoring solution that lets you run real PromQL on logs. But there will be similar ways of accomplishing the same alerting logic.
For local development, yes. But I see that almost never done in production settings, especially with containerized workloads. Is there a neat way to do it in Kubernetes?
I loved this article. I am working on an LD_PRELOAD object for some observability usecases, and while there was a learning curve, Zig an its lack of dependence from LibC is a breath of fresh air!
Adopting OpenTelemetry does not have to be hard for common use-cases. On Kubernetes, the Dash0 operator (https://artifacthub.io/packages/search?repo=dash0-operator) automatically instruments Node.js and Java workloads (and soon other runtimes) with just a custom resource created in a namespace. It works with all OpenTelemetry backends I know of.
Disclaimer: I am one of the authors of the Dash0 operator and work on Dash0 (https://www.dash0.com/), an OpenTelemetry-native observability platform.
I am certainly biased here because OpenTelemetry and Prometheus have been at the core of my professional life for the past half decade, but I think that the biggest challenge, is that there are many different ways to get you to a good setup, and people get lost in the discovery of the available options.
True, but that approach is also limited in what it can autoscale on, namely Utilization metrics. They are not always good data to make scaling decisions on (covered in the article).
I looked into how to wire the Horizontal Pod Autoscaler for Kubernetes to Dash0, the OpenTelemetry-native observability tool I am working on, and it turned out to be surprising simple.
We explicitly test 1.28+. But I have in front of me a 1.27.13 and the operator works on it without issues. And probably goes way further back than that.
What is the oldest version of Kubernetes you have seen in the wild recently?