I completely agree w/ your points about why observability sucks:
- Too much setup
- Too much maintenance
- Too steep of a learning curve
This isn't the whole picture, but it's a huge part of the picture. IMO, observability shouldn't be so complex that it warrants specialized experience; it should be something that any junior product engineer can do on their own.
> I can definitely imagine having Claude debug an issue faster than I can type and click around dashboards and query UIs. That sounds fun.
You're not the first person I've met that has articulated an idea like this. It sounds amazing. Do you have an idea about why this approach isn't broadly popular?
This is a super insightful comment & there is a bunch that I want to respond to but I can't do it all neatly in one comment. Hahaha
I'll choose this point:
> reliability is still ultimately an incentive problem
This is a fascinating argument and it feels true.
Think about it. Why do companies give a shit about reliability at all? They only care b/c it impacts bottom line. If the app is "reliable enough" such that customers aren't complaining and churning, it makes sense that the company would not make further investments in reliability.
This same logic is true at all levels of the organization, but the signal gets weaker as you go down the chain. A department cares about reliability b/c it impacts the bottom line of the org, but that signal (revenue) is not directly and attributable to the department. This is even more true for a team, or an individual.
I think SLOs are, to some extent, a mechanism that is designed to mitigate this problem; they serve as stronger incentive signals for departments and teams.
Hello! Yes, you are right - observability and APM have both been around for many decades, but the incarnations that most people are familiar with are the ones that emerged in the 2010s.
My intention wasn't for this post to be a comprehensive historical record. That would have taken many more words & would have put everyone to sleep. My goal was to unpack and analyze _modern observability_ - the version that we are all dealing w/ today.
Wow, this is great. I have been using Inngest for over a year now and really like the APIs you guys have created for defining step functions / event handlers. I'm very glad to see that this API is now open-source so that it can be adopted more broadly!
I've been programming for 15 years now, 10 years professionally, and have worked at 5 startups. These are my reflections on the first decade of my career.
Also, an announcement: I’m starting something new!
I don't have much experience w/ "vanilla" LangChain or the LC Python tools, but I've been an avid user of Deployments and the LangGraph TypeScript SDK for like a year now (started using Deployments back when it was still called "LangGraph Platform"). I think I might be one of the oldest users of both...
To be honest, my first impressions were mixed. The Deployments product was very early/new and rough around the edges (bad monorepo support, for example). And the LangGraph TypeScript SDK felt... not super TypeScript-y. (I get the sense they ported over a bunch of abstractions from the Python package).
But the benefits outweighed the costs. In September 2024, Deployments + the LangGraph TypeScript SDK was one of the only ways that you could say FOR SURE that your agent was going to (a) work and (b) run smoothly in production.
(Also, their team worked hard as hell to ramp myself and my teammates up on agents. I was really won over by this and will be a LC stan forever as a result.)
Over the past year, I've seen both products evolve and mature significantly - to the point where all of those initial problems seem to have been addressed.
I still feel that the overhead associated with building and hosting your own agent from scratch is too much for most teams to take on, even at large companies. (In particular, first-class support for streaming is massive; this would be a huge PITA to build in-house.) I'm happy to let LangChain take care of all of this for me.
Overall, I've been very impressed at LC's ability to iterate and absorb feedback - especially when some of it is... not delivered in the kindest way. They're a very humble and hardworking team. Makes me happy to see them winning.
I love the idea that typed languages create backpressure. I have very little confidence in peoples' ability to create well-designed (in software architecture terms) projects.
My company is in the process of building out an integration test suite for our microservices platform. A few pain points:
1. The tests themselves are housed in a separate repository, so you can't update the tests alongside your service. This means every change to a service has to be backwards compatible. Hello, multi-step rollouts.
2. Our environments must be highly configurable, so that every permutation of versioned services can be integration-tested. This is forcing us to adopt an unnecessarily complex container orchestrator.
3. Service owners are not excited about setting up and contributing to yet another project. We end up with a lot of out-of-date integration tests. Lots of noise, if you ask me.
I think a combination of contract testing (e.g. using Swagger, Pact) with monitoring, canary deployments, and automatic rollbacks would be easier to maintain and just as effective at catching bugs.
Postman can be handy for quick and dirty HTTP tests. Unfortunately, I find the Postman GUI unintuitive. The alternative -- working with minified JSON -- is equally painful.