Virtual Log-Structured Storage for High-Performance Streaming [pdf](hal.inria.fr)
hal.inria.fr
Virtual Log-Structured Storage for High-Performance Streaming [pdf]
https://hal.inria.fr/hal-03300796/file/virtual_log_KerA30072021.pdf
3 comments
Your dream has arrived! Redpanda [1] is a non-Java, ZooKeeper-free streaming platform. It's Kafka compatible, so you can use any Kafka client, of which there are many.
Finally, and perhaps most importantly, it is substantially faster than Kafka.
Disclosure: I work there.
---
[1] https://github.com/vectorizedio/redpanda
Finally, and perhaps most importantly, it is substantially faster than Kafka.
Disclosure: I work there.
---
[1] https://github.com/vectorizedio/redpanda
The problem with Kafka is not Java nor ZooKeeper - it is that non-native (non-Java) clients are buggy as hell (last time I touched them), because everybody have to re-implement all the complex logic for each language (often multiple times).
Do you have any plans for Redpanda-native protocol/client library?
Why non-java? What do you think is missing that you'd like to build?
Easier to use? Kafka is rather trivial to setup and scale to any size, even more so as they move away from zookeeper
Can't think of a more well behaved java app. Heap usage generally low, never seen GC issues and fast as needed
Can't think of a more well behaved java app. Heap usage generally low, never seen GC issues and fast as needed
Is this for write heavy workloads instead of ready heavy workloads? Some of the ideas seem similar to Virtual Consenus in Delos [1].
[1] https://blog.acolyer.org/2020/11/09/delos/
[1] https://blog.acolyer.org/2020/11/09/delos/
Source code here: https://gitlab.inria.fr/KerA/kera
hopefully at some point a more easy to use, non java, "cloud-native" distributed log with decent client apis shows up. (if anyone wants to help building it, please ping me. i am 10 years into building startups and companies with architectures ontop of kafka and pulsar)