Building Self-driving Kafka clusters using open source components(slack.engineering)
slack.engineering
Building Self-driving Kafka clusters using open source components
https://slack.engineering/building-self-driving-kafka-clusters-using-open-source-components/
18 comments
You're conflating Apache Kafka, the distributed log and Kafka Streams, the streaming library built by Confluent that leverages Apache Kafka.
Kafka itself is straightforward to grasp, Kafka Streams, yep, your concerns sound familiar.
Kafka itself is straightforward to grasp, Kafka Streams, yep, your concerns sound familiar.
Kafka Streams is a core part of the open-source Apache Kafka project.
You may be mistaking it for kSQL, or ksqlDB as they call it now, which is a Confluent project built on top of Kafka Streams.
You may be mistaking it for kSQL, or ksqlDB as they call it now, which is a Confluent project built on top of Kafka Streams.
Be that as it may, I agree with the spirit of the grand-parent comment, which is that if you stick to straightforward Pub/Sub Kafka works great.
It is, but it's ancillary to Kafka the distributed log.
Have you tried nats.io and it’s new streaming tech, Jetstream? Night and day difference in terms of management and ease of use. In my last org I was fighting so hard against folks wanting to adopt Kafka, and then I found out about nats, total game changer
Or Redpanda. We like to think it's Kafka but better.
Obvious bias: I work for Redpanda
Obvious bias: I work for Redpanda
NATS is open source and CNCF. Redpanda is more a SaaS -- if it goes away, so does your app.
https://github.com/redpanda-data/redpanda
You're welcome to use it yourself and not pay us a dime. We're a business so we need to monetize but our core is to build a better Kafka. The nice thing is that if you've already invested in building against Kafka but you're exhausted by running Kafka, ours works better and faster. (Again, I'm biased so don't trust me, verify.)
You're welcome to use it yourself and not pay us a dime. We're a business so we need to monetize but our core is to build a better Kafka. The nice thing is that if you've already invested in building against Kafka but you're exhausted by running Kafka, ours works better and faster. (Again, I'm biased so don't trust me, verify.)
I don’t think that holds here, Redpanda has a Kafka compatible API layer and you can 100% self-host it. Being CNCF or not doesn’t seem like a meaningful or useful distinction here.
The compatibility isn't 100% there yet.
NATS and Kafka offer different semantics.
By default, NATS offers non durable messages and different delivery options. I believe you could get durable messages, but last time I read up on this, it required running extra features or infrastructure to support this.
By default, NATS offers non durable messages and different delivery options. I believe you could get durable messages, but last time I read up on this, it required running extra features or infrastructure to support this.
You now get durability on whatever subjects you want via NATS Jetstream: https://docs.nats.io/nats-concepts/jetstream
No additional infrastructure, just enable it on the nodes you want persistence on and create a stream. Publishers don't even have to change, consumers have a great api with lots of options
No additional infrastructure, just enable it on the nodes you want persistence on and create a stream. Publishers don't even have to change, consumers have a great api with lots of options
Oh that’s pretty cool, I wasn’t aware of that! Nice to that functionality added!
Same experience here. We're using Kafka Streams and it's a horrible experience due to its unreliability. We need a full-time person just to make sure it stays up.
Same experience here. The only thing excellent about Kafka is marketing. It is endless, relentless and everywhere.
I'm confused about the breezy way in which they distribute partitions and consumers. Since the producer computes the partition key and the consumer chooses which partitions to bind, how does the central kafka team enforce balance?
One would need some sort of policy about how developers use the Producer and Consumer API's for these things, but that would be a very difficult thing to manage and enforce I would think.
One would need some sort of policy about how developers use the Producer and Consumer API's for these things, but that would be a very difficult thing to manage and enforce I would think.
Their upgrade strategy is really awful. Kafka can do rolling upgrades, it's one of the few things that has worked reliably well for me in Kafka over the past 6 tears. Pretty much as simple as a broker reboot and setting inter.broker.protocol.version/log.message.format.version correctly.
They made it way harder than it needs to be.
They made it way harder than it needs to be.
I’ve been writing software for a living for more than two decades, and Kafka is the most horrible, impossible-to-grasp boat anchor I’ve ever had to deal with.
I hate every minute of my working day that I have to deal with Kafka Streams’s hissy fits.