Personally, I'm happy to see them evolving it with my juices left intact. When we transitioned benthos.dev to https://docs.redpanda.com/redpanda-connect we gained a much larger (and more competent) docs team, but as a consequence had to trim a lot of the old personality. I don't regret the transition but I do take a bit of comfort seeing the WarpStream peeps continue to have fun with it.
It's like shedding an almost intact full body skin and finding comfort in watching colleagues take it bowling, having a great time.
Sure, if they change the MIT license of the core engine then you could fork it at that point. What they're doing right now is taking on a much larger maintainence burden than potentially necessary and fragmenting the ecosystem at the same time.
You're also at the same risk if you choose to use their fork.
It absolutely doesn't need a fork. The entire project is designed specifically to allow vendors and users to have their own ecosystem of plugins and they can all compile and integrate seamlessly. I'll be explaining live in 30 mins: https://www.youtube.com/watch?v=X8nVdUuWZ80
Hey it's Ash (the maintainer being talked about in the blog). I'm not one for fork drama and I haven't had a chance to fully read the blog so I don't have a lot to say. However, this is a full fork of the entire codebase, which means plugin authors will need to choose one project or the other and are locked in, and is entirely unnecessary on both a technical and legal perspective.
If they'd instead chose to fork the plugins themselves (the only parts where the licenses changed, all except two are Apache V2) then all users can pick and choose which ones they include in their projects, and it doesn't fragment the ecosystem at all. Your plugins would compile in my project, and mine would compile in yours.
The part they're choosing to fork here, which will cause this rift in the community, is still MIT licensed: https://github.com/redpanda-data/benthos. If they simply chose to continue using this MIT part we can all live happily together in a utopian society fully saturated with plugged blobbery.
Edit: I'm bit a baby brained so I forgot that I'm literally streaming live in 30 minutes in order to explain all the changes in detail for those out of the loop: https://www.youtube.com/watch?v=X8nVdUuWZ80
Yeah that's my bad, the site assumes you already know what benthos is as it's an early stages UI for it. The best place to start is https://www.benthos.dev, or if you like dumb videos: https://youtu.be/88DSzCFV4Ng
If I want to achieve something then I find the joy in pursuing it, at which point I will naturally do it when I'm capable and will do other things when I'm not.
Letting my attention float around freely (within limits obviously) often looks like I have a low attention span. However, with a birds eye view on what I'm able to get done I look like I'm hyper focused.
And if I can't find the joy in a long term pursuit then it's very unlikely to be worth my time.
Hey everyone, this is a video I put together summarising a decades worth of stream processing delivery guarantee misconceptions and bugs that I've seen frequently.
I'm not trying to scare anyone away from stream processing, in fact a lot of the issues outlined also apply to automated batch processing systems as well. Personally, I think that being realistic and pragmatic about failure conditions makes these systems less intimidating.
Anecdotally I'd say professions that often double up as a hobby and don't have a significant material or energy cost to exercise will be more susceptible to burn out. If you're _really_ into software engineering and struggle to pace yourself then burn out will often stop you sooner than finger fatigue or your computing costs.
I'm not saying workplace pressure etc isn't a factor, but those are also similar factors in pretty much all careers.
Maintaining an OS project definitely gives you a crash course in (and boosted respect for) non-engineering roles at tech companies.
However, I'd say if you're maintaining something and don't enjoy one or more of these hats then keep in mind that totally dominating a problem space doesn't need to be the end goal. There is such a thing as a lifestyle OS project and you can still make a success out of it, gain a loyal (perhaps modest) community, and enjoy taking it slow.
I'm massively rooting for this. I really wanted to use Gitter as the official chat for my projects, but the experience was just so painful that I ended up fragmenting the community across Discord and a gated Slack. I would love to have a forever home where I can feel comfortable getting everyone on board.
Still, assuming there are no other edge cases there, it doesn't address the other problem where a hypothetical consumer of the output topic is reading an at-least-once feed of your exactly-once topic. In order for that not to be the case then the consumer must also be idempotent, in which case what value was gained from the original deduplication?
Yeah but the first time it was read the key was stored in RocksDB, so the second time it gets consumed after the crash:
"If the message already exists in RocksDB, the worker simply will not publish it to the output topic and update the offset of the input partition, acknowledging that it has processed the message."
It gets dropped as if it were a duplicate, oops!
Edit: I reread the relevant section...
"If a message was found in the output topic, but not RocksDB (or vice-versa) the dedupe worker will make the necessary repairs to keep the database and RocksDB in-sync. In essence, we’re using the output topic as both our write-ahead-log, and our end source of truth, with RocksDB checkpointing and verifying it."
Not sure entirely what they mean by "(or vice-versa)", if the message exists in RocksDB but not in the output topic how you distinguish between a real duplicate and a crash artifact?
If the last message of a crash happens to be a real duplicate and this recovery mechanism reintroduces it into the pipeline then you have a duplicate.
Either way, it's not an exactly-once feed. At best (assuming message loss isn't possible) it's an at-least-once feed that usually appears to be exactly-once.
"What’s more, we want to ensure the information about which events we’ve seen is written durably so we can recover from a crash, and that we never produce duplicate messages in our output."
Your processor is described as writing from Kafka to Kafka and using a persisted RocksDB instance to check message identifiers. How then do you ensure messages aren't dropped if your processor crashes or gets killed after checking against RocksDB but before the message is flushed to the Kafka broker?
Also is your producer writing to Kafka not at-least-once? If so then even if it removes all duplicates in its processing stage the feed written to your output topic could still contain duplicates.
By contrast deduplicating on consumption avoids that problem entirely by attempting to build an idempotent consumer, which results in an exactly-once. Although in this case they have identified edge cases of duplicates they're comfortable with.
"amounts to 1.5% of its worldwide turnover in 2017"
I imagine that's a significant sum but I'm struggling to get my head around it. If so then good for the ICO I suppose. I remember reading endless comments a few years back speculating GDPR would never have any bite.
Processor implementations are able to carry their own state, or share state across worker threads or deployments using their own mechanisms, but Benthos doesn't provide any tooling for that. None of the processors you get out of the box need computational state. You currently get an ALO stream (provided you use ALO protocols), vertical & horizontal scaling as per your config, and any glue you need between services.
It would be a nice stretch goal to have standard tooling within Benthos to share distributed state, perhaps with some ability to do exactly-once processing, but that's not the focus of the project right now.
All Benthos processors are stateless, except for a few such as batch, combine, etc, which do basic tasks such as building up batches from consecutive messages. It's possible to create sliding window processors on top of Benthos, but there aren't any general tools for doing that yet.
It's like shedding an almost intact full body skin and finding comfort in watching colleagues take it bowling, having a great time.