I'm implementing a Kafka/Kinesis-like append-only commit-log streaming. My goals are to implement a streaming-engine that runs with a single binary, easy to distribute, no need for zookeeper, easy to host/operate, with pull-based API, similar to Kinesis, where the consumers are not registered and the consumers manage their own cursors, among other things.
https://github.com/14-bits/voik
At this point, I have built the main parts of the commit-log, I'm trying to follow this series of articles for insights on what to build first... as soon as I feel the storage mechanics is good I'll start looking into data replication and delivery.
I would really appreciate contributions both to the code from a Rust perspective (I'm quite new to the language), as much as from system's perspective and such.
The project has a small roadmap with items that I'm working, but if people are interested in collaborating we could probably move those to more meaningful issues for discussion purposes.
There is also a lot of tooling/CI/documentation and community related topics that need attention, so if you feel like participating but is not an expert, it's a good place to start.
I'm implementing a Kafka/Kinesis-like append-only commit-log streaming. My goals are to implement a streaming-engine that runs with a single binary, easy to distribute, no need for zookeeper, easy to host/operate, with pull-based API, similar to Kinesis, where the consumers are not registered and the consumers manage their own cursors, among other things.
https://github.com/14-bits/voik
At this point, I have built the main parts of the commit-log, I'm trying to follow this series of articles for insights on what to build first... as soon as I feel the storage mechanics is good I'll start looking into data replication and delivery.
I would really appreciate contributions both to the code from a Rust perspective (I'm quite new to the language), as much as from system's perspective and such.
The project has a small roadmap with items that I'm working, but if people are interested in collaborating we could probably move those to more meaningful issues for discussion purposes.
There is also a lot of tooling/CI/documentation and community related topics that need attention, so if you feel like participating but is not an expert, it's a good place to start.