I set up a fairly large Prom-based architecture which I later on migrated to VictoriaMetrics (VM) so I think I can chime in here.
Both Prom and VM are exceptionally stable in my opinion, even on _very_ large scales. There were times when I had a single (Prom, later VM) and not-overly-large instances scrape 2Mio samples/s without any issues. In addition to fairly spiky query loads.
However, if something does go wrong, the single most impactful difference between VM and Prom is simply the difference in startup time. Prometheus with 2TB of metrics takes _forever_ to start up. We're talking up to 2 hours on SSD while VM just... starts.
Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it.
The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense.
I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.
I mentioned it in another reply, but https://perses.dev/ is probably the most promising alternative.
Besides that, if you're feeling masochistic you could use Prometheus' console templates or VictoriaMetrics' built-in dashboards.
Though these are all obviously nowhere near as feature rich and capable as Grafana and would only be able to display metrics for the single Prom/VM node they're running on. Might enough for some users.
I only know of https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.
I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries.
I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching.
Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.
I wish the big providers would offer some sort of trial period where you can evaluate models in a _realistic_ setting yourself (i.e cli tools or IDE integrations). I wouldn't even mind strict limits -- just give me two hours or so of usage and I'd already be happy. Seriously.
My use-case is probably pretty far from the usual tasks: I'm currently implementing a full observability platform based on VictoriaMetrics / Victorialogs + Grafana. It's quite elaborate and has practically no overlap with the usual/cloud solutions you find out there. For example, it uses an authenticated query stack: I use the Grafana oauth token to authenticate queries by injecting matchers via prom-label-proxy and forward that to promxy for fan-out to different datasources (using the label filter to only query some datasources). The IaC stuff is also not mainstream as I'm not using any of the big cloud providers, but the provider I use nonetheless has a terraform provider.
As you can imagine, there's probably not much training data for most of this, so quality of the responses varies widely. From my experience so far Claude (Sonnet 4.5 ) does a _much_ better job than GTP-5 (Codex or normal) with the day-to-day task. Stuff like keeping documentation up to date, spotting inconsistencies, helping me find blind spots in the Alerting rules, etc. It also seems to do better working with provided documentation / links.
I've been using Claude for a couple of weeks now but recently switched to codex after my subscription to Claude ran out. I was really curious after reading a lot of good things about it but I gotta say, so far, I'm not impressed. Compared to Claude it gives wrong answers much more frequently (at least in this domain). The results it produces take much more effort to clean up than Claude's. Probably on a level where I could just invest the time myself. Might be that I do not yet know how to correctly prompt GPT but giving both tools the same prompt, Claude does a better job 90% of the time.
Anyway, I guess this is my long-winded way of saying that the quality of responses "off the beaten track" varies widely and is worth testing several models with. Especially if your work is not 70+% of coding. Even then I guess that many benchmarks have seized being useful by now?
Correct. This can be disabled [0] but you need to work around this then. Usually you can "just" use host-networking and manage iptable rules manually. Not pretty but in that case you at least know what's done to the system and iptables.
We used Seafile in $old_job a couple of years back. Still had some rough edges back then (mostly with user management) but the rest was rock solid. Loved working with it, managing and sharing files was pretty easy.
I'm not a aware of any other full-fledged solution. There is some opinionated tooling written in Python [0] and Go but nothing coming close to defined.net. This is definitely a point where other overlay solutions look way better.
After having searched (and implemented) this myself for work, the only practical solutions I found were 1) smallstep [1] or 2) Terraform (with the nebula provider [2]) and a CM tool of your choice. The latter can be nicely combined with the ansible provider if that's your CM of choice.
Nebula is fantastic, absolute love it. We use it in production. Cert management can be a bit of a pain on a large scale but there's an excellent Terraform provider [0] that can help. Coupled with the Terraform ansible provider and a little bit of scripting you can automate anything related to cert provisioning and renewal.
You should give nebula a try. I've recently switched my private VPN setup from wireguard to nebula and am looking into using it for work. It has some really nice features (for our use case), so ymmv. But so far it's been fantastic and very easy to use.
Yeah, I think many "buy it for life" items are generally only sensible if you depend on them either for work, a somewhat regular hobby, or for everyday use.
Sometimes it almost hurts me to buy cheap junk. But then I also realize that this is perfectly adequate for my use cases and even $10 more for another product would not be worth it. (And who says that the other product would really be better?).
I'd rather have some physical knob that I can use to turn something off, even if I have to bust out the pliers. Good luck shutting off some touch control that's on the fritz.
The stove at my parents place has some touch control to activate another heating ring(?). Not only is that hard to activate with wet or greasy fingers, but after a couple of years of use it keeps turning itself off now. Thank god that thing fails "closed" and does not (yet?) spontaneously activate effectively doubling the heating power.
This is one of the reasons I bought a USB hub were I can switch connected devices off and on (disconnects their power lines). If you're lazy like me you can plug another USB hub into one of the ports so that can switch multiple devices (webcam and mic) with one button press instead of two :)
Searching for "usb hub power switch" should get you a few hits.
Both Prom and VM are exceptionally stable in my opinion, even on _very_ large scales. There were times when I had a single (Prom, later VM) and not-overly-large instances scrape 2Mio samples/s without any issues. In addition to fairly spiky query loads.
However, if something does go wrong, the single most impactful difference between VM and Prom is simply the difference in startup time. Prometheus with 2TB of metrics takes _forever_ to start up. We're talking up to 2 hours on SSD while VM just... starts.