What I like using Grafana Loki for (and where I avoid it)(utcc.utoronto.ca)
utcc.utoronto.ca
What I like using Grafana Loki for (and where I avoid it)
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLokiWhatILikeItFor
7 comments
Oh interesting. I like that qryn exposes Grafana compatible query APIs. I’ve been playing with SigNoz (another Clickhouse-based, OTel compatible project for ingesting and analyzing logs, metrics, and traces) and I wish it would offer the same capability.
Thanks for the feedback! qryn developer here, and that's exactly our design goal: requiring no custom agents or clients in order to use our solution, through API emulation. There's a chance you could use qryn as a reader for data written by other clients, and that's exactly what we do with opentelemetry where we have a qryn-otel-collector allowing direct writes into ClickHouse at max speed, just like signoz, and reading through our Grafana compatible APIs using qryn.dev or qryn.cloud
But there are no straight up syslog/journald adapters?
qryn and vector get along very well! We use it all the time for testing and developing qryn and qryn.cloud and most of our users love it! But since qryn supports Logs, Metrics and Telemetry at once, we're just as compatible with any Agent supporting Loki/LogQL, Influx protocol for metrics and logs, Elastic Bulk, Prometheus for metrics, opentelemetry for everything... and more coming!
Feel free to open an issue on our repository if you end up trying it and/or would like us to help out!
https://qryn.dev https://qryn.cloud
Feel free to open an issue on our repository if you end up trying it and/or would like us to help out!
https://qryn.dev https://qryn.cloud
Thank you. In this case,it was more a reference to:
> Our central syslog server is a central source of accessible, text based truth. It requires almost no infrastructure to be working on any machine in order to get logs to it and accept logs (...)
From the linked post on central log server (and Loki):
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLok...
Unfortunately, AFAIK journald and qryn don't speak rsyslog - so you'll need something "alien" - all though vector seems very straightforward.
> Our central syslog server is a central source of accessible, text based truth. It requires almost no infrastructure to be working on any machine in order to get logs to it and accept logs (...)
From the linked post on central log server (and Loki):
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLok...
Unfortunately, AFAIK journald and qryn don't speak rsyslog - so you'll need something "alien" - all though vector seems very straightforward.
A couple years ago we were hurting from having no easily accessible centralized logs. We also had some in-house timeseries database tech that we were looking to deploy more widely but needed first to stress test.
After stumbling on an overview of Loki, we whipped up a service that presented a Loki-compatible API on top of our timeseries database, and killed two birds with one stone. Because we have full control over the data ingestion and storage too we are able to keep the data in a structured form with full type information.
It's fantastic in that it ties in with the rest of our use of Grafana too.
After stumbling on an overview of Loki, we whipped up a service that presented a Loki-compatible API on top of our timeseries database, and killed two birds with one stone. Because we have full control over the data ingestion and storage too we are able to keep the data in a structured form with full type information.
It's fantastic in that it ties in with the rest of our use of Grafana too.
Loki is great but PromQL is a doozy. I’m basically useless on grafana without the query builder. Tempo is also pretty cool if you’re emitting otel spans.
Loki is the bomb-dizzle when it comes to real-time monitoring. First it’s designed to provide real-time monitoring of log data, allowing users to detect issues and troubleshoot problems as they occur. Gonna come in handy, particularly for mission-critical applications where downtime can have severe consequences.
Single node kube cluster running Loki and promtail and I’ve had great results until I started trying to push in some bare metal machine logs which cause 503 from nginx ingress. Then Loki stops working in Grafana.
But inside the cluster it’s great. Though my Loki helm chart is still on v3 rather than 4
But inside the cluster it’s great. Though my Loki helm chart is still on v3 rather than 4
We are hosting the grafana stack at my work. And if there are some comprehensive articles about trying to keep metrics counts low, and hosting on block storage with s3. I would appreciate peoples anecdata and links.
i am looking into loki to use at work. Just wondering do most of you self-host or use the grafana cloud?
AFAIK logshipping with journald is bespoke - but vector should be able to handle both syslog and journald - for relatively painless setup ...
Not sure if tailing is better with qryn though.
https://github.com/metrico/qryn
https://vector.dev/docs/reference/configuration/sources/sysl...
https://vector.dev/docs/reference/configuration/sources/jour...
https://www.digitalocean.com/community/tutorials/how-to-cent...