Jared: "We are sending out a proposal for Stacked Diffs on
@GitHub
to trusted design partners to gather initial feedback over the next few days. From there we’ll iterate and share the gameplan"
Like many developers, we've built our fair share of workflows that export data to 3rd-party services. They always start simple: pull data, hit an API, job done! Then the problems show up. We hit API limits, services go down, and those quick-and-dirty workflows become a major source of headaches.
The knee-jerk reaction is often to add a queue! Sure, it helps for a while. But queues introduce their own complexity: handling failures, managing retries, creating visibility... It's a band-aid, not a cure, and we've been wrestling with this problem for too long!
In this blog post, we'll break down:
- Why queues fall short when building truly resilient integrations
- The core principles behind building scalable, fault-tolerant async workflows
- Practical techniques that go beyond the limitations of queues
If you're done with fragile systems and want to level up your integration game, this one's for you!
We've had enough of traditional orchestration frameworks. That's why we created dispatch.run, aiming to streamline coding by integrating resilience more naturally.
The core of our solution? Distributed Coroutines. These aren't your typical tools; they're designed to enhance flexibility and reduce complexity in distributed systems.
We've detailed our approach and the potential of Distributed Coroutines in a new blog post. It's about making development smoother and more intuitive.
Let's discuss the future of distributed computing.
Forking Go means that you have to maintain a copy. That's a lot of work. Instead, the idea is to implement coroutine as a library and work toward an integration with the Go stblib.
The idea is that we are running few multi-tenant Kafka clusters to amortize the operation cost. These clusters host many topics with different topology depending on different requirements, cost profile and scale.
Some clusters are hosting hundreds of topics, thousands of partitions and used (consumers/producers) by hundreds of services.
I guess we just haven't got the need to use ACL yet. This may be something we will look later on. We have been started to implement what's needed to leverage things like SASL in our instrumentation but we are not there yet (see https://github.com/segmentio/kafka-go).
Yes, we looked at Terraform to manage our topics. We are big Terraform user to manage our infrastructure. We mostly didn't go that way because we didn't want to have to manage the Terraform state. We also wanted the tool to be extensible and writing custom TF providers can sometime be tricky.