It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to.
As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.
That's why their entire business model -- like Astronomer's -- is geared toward cloud hosting. The architecture is so complex it takes a full time team to run it.
Have you looked into DBOS? Same thesis: durable and reliable workflows are hard to manage -- it just doesn't have to be as hard as Temporal makes it be :)
I've talked to dozens of engineers who built their home grown "durable" stack. Most of them eventually moved on to buying vs building, when their system actually scaled. It's just not a side-hustle to build a foundational reliability layer.
Yeah, we've observed that too: people start implementing their own retry logic, idempotency, etc. But then they grow a hard to maintain, complex stack that's not their core business logic. There's a reason why there is a dedicated team building DBOS, every day. Because it's not that easy to build a solid durable workflows engine on Postgres.
As you said, the example is simple and it might not be obvious to people without prod experience what the problems can be. Postgres can give you all the primitives you need to solve this at the application layer. Durable workflows on Postgres is an effective way to access these primitives.
We've found that durable workflows is a much needed primitive for agents control flow. They give a structure for deterministic replays, observability, and, of course, fault tolerance, that operators need to make the agent loop reliable.
I notice you didn't provide any specific comparison alongside that comment, which makes me feel frustrated because I think the Temporal workflow SDK is very different. Architecturally, Temporal and DBOS are at two opposites of the durable execution spectrum. I'd love to understand what makes you think this work is a mere copy and paste. Would you be willing to share some more with me?
Thanks for the comment (author here). I wanted this post to focus on the Golang specific implementation, not dwell on the durable execution ecosystem at large.
With respect to context, I don't know that anyone invented "having their own context". Go interface are extendable and pretty much every major framework I know of implement their own context.
Would love to learn more about the gaps that offset you. We're constantly improving here ;)
Conductor is about enterprise features like automatic workflow recovery, alerting, or RBAC. The GUI is a nice to have -- but all your workflow data are in Postgres. You can access it very easily.
Durable execution has already been mentioned as the existing solution for this problem, but I would like to call out a specific pattern that DE makes obsolete: the outbox pattern. Imagine just being able to do do
send a()
send b()
And know both will be sent at least once, without having to introduce an outbox and re-architect your code to use a message relay. We can nitpick the details, but being able to "just write normal code" and get strong guarantees is, imo, real progress.
The hype is because DE is such an dev exp improvement over building your own queue. Good DE frameworks come with workflows, pub/sub, notifications, distributed queues with tons of flow control options, etc.
website: http://www.maxdml.com/