I also think that printf debugging is fine, but there are better tools [1] - however they might be more involved to setup, or not be available in your codebase.
Go: go templates and fiber for http serving. I have a mini framework for adding some structure to the templates (folder per feature: contains controller, templates, etc, and some naming conventions)
If I understood correctly, you can achive this with `hx-target` and `hx-select` - I'm doing this in an app to replace an editor content based on which link is clicked.
Yes, and itel has instrumentation libraries which do this.
However, no automatic instrumentation can do everything for you; it can't know what are all the interesting properties or things to add as attributes. But adding tracing automatically to SQL clients, web frameworks etc is very valuable
As other posters have mentioned, the incument companies rebranding to Observability definitely are expensive, because they are charging in the same way as they do for logs and/or metrics: per entry and per unique dimension (metrics especially).
Honeycomb at least charges per event, which in this case means per span - however they don't charge per span attribute, and each span can be pretty large (100kb / 2000 attributes).
I run all my personal services in their free tier, which has plenty of capacity, and that's before I do any sampling.
Yes, but going to this effort, why not move to tracing instead?
A migration path I could see might be:
- replace current logging lib with otel logging (sending to same output)
- setup tracing
- replace logging with tracing over time (I prefer moving the most painful areas of code first)
Agreed the amount of stuff showing in stack traces can be annoying - but it does depend on language.
I agree 100% in javascript/typescript its annoying, and I would love to get rid of them, In go however, there isn't an extra stack frame. Nor in C# thinking about it.
The config file of functions to trace is a really interesting idea. How would you handle wanting to add data to the spans from inside those functions though? e.g. I want to add all kinds of props to the spans that can't be known except inside the function executing.
The only thing I can recall being a bit of a stumbling point at the beginning was finding "my" trace from a local environment. I just didn't know where to find it. Once I figured that out I've not had problems.
The high-level view and being able to draw a box around something that looks weird on my graph, and honeycomb tells me what's different inside and outside the box is amazing (called bubble up, if you're searching).
It's faster than any other tool I've used; usually data is available by the time I've switched to their ui from the curl command to our API. It's mind blowing actually.
Other saas and self hosted options I've tried have all been awful in some way or other; honeycomb is a breath of fresh air, and going back to other tools after using honeycomb is painful.
I'm not sponsored or working for them btw, they just make one of the few products that I genuinely love using.
Only read the flag once! For multiple flags we tend to have the first step in the build read all flags we care about and export them to environment variables.
Sure you can use the environment variables, and if you have simple uses cases this is probably the best way to go.
In this case, we're using LaunchDarkly's targeting to serve different variations to groups of users, and also using the in built monitoring LaunchDarkly provides to see how often each is requested etc.
> Deploy is the act of moving software into an environment and running it.
> Release is the process of making a feature visible to a user or subset of users.
In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job.
If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.
I understand that, but would counter with that it sounds like deploy == release, and that if that weren't the case, you could deploy more often.
However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.
I like to do short lived credentials using Vault (e.g. vault can create say db access credentials dynamically), but for things like API keys where I can't do that..? Is the Vault KV store the source of truth?
[1]: https://andydote.co.uk/2024/11/24/print-debugging-tracing/