14 years in work experience and 21 years in commercial software development, focused on engineering and science fields such as databases, compilers, deep learning, language design, high-performance computing, high-concurrency backend and network protocol and communication from data centers, to clouds and to decentralized internet.
MQTT and Kafka are different things. MQTT is not necessarily better or worse than Kafka, vice versa.
But, sadly, people are often caught in their own loops, when there are or can be created better options.
I have created a new, free, single binary data-service platform for IoT, JoinBase: https://joinbase.io/
This single binary data-service platform has proved:
1. Kafka is not more suitable for industry or higher performance than MQTT, even from the protocol level
With carefully crafting, JoinBase has saturated one PCIE 3.0 NVME sustained write bandwidth (25 million msg/s) in single modern node. This is, in fact, can not been done by the Java-based Kafka.
2. Use MQTT and Kafka together is unnecessary.
This only makes your pipeline more complex, expensive, but much unstable and slower.
JoinBase can do arbitrary message preprocessing and auto-view(WIP). Streaming does not have to be owned through a separate monster.
3. High performance or ease of use, has nothing to do with the size of the software if the product can be properly engineered.
5MB Single binary JoinBase is enough to beat many monsters in the IoT/AIoT data pipleine:
+ sustained batch MQTT message write throughput: ~10x faster than Kafka and ~5x faster than that of one popular broker
+ basic SQL analytics: 3-4x faster than ClickHouse
+ HTTP interface concurrent queries: ~100x higher than ClickHouse
+ ...
More could be seen in our 2022 summary blog: https://joinbase.io/blog/joinbase-2023/
There are historical reasons for all of these, of course. But it could be great that we break out of own mindset loops.
JoinBase shares some basic ideas with PostgREST, but we provide a much simpler solution for the view of database: installation, configuration, authentication, schema creation and even performance are damn simple within JoinBase.
Different to PostgREST, the HTTP interface of JoinBase is integrated in the database. So, no verbose setup, one binary rules them all!
Our HTTP interface is inspired from the ClickHouse, but we provide 100x message throughput than that of ClickHouse in the HTTP interface. JoinBase's HTTP interface is so fast that you can use it to provide unlimited production-level REST services without any worry.
"Third parties to help" is a good point. Although under the USAL, you can make "third parties" to become users.
Thanks for the suggestion! The USAL is new, it may be updated to allow pluggable additional grants to allow more flexibility, for example, if the licencor as a business entity does not exist, then the licensed works could be changed to another license.
Copyleft does not solve the problem: how to build a positive business cycle. No company seriously contributes to the copyleft. That is why the Apache License is raising. And you can see that all changed licenses mentioned in the article are not copyleft. Copyleft is great for licencors, but most of companies in the world, does not want to share their customs. This is the businesses.
The Linux kernel is almost the only one in the world and cannot be copied.
But the Apache License is truly dead for startups. Most of "open source" startups are living on the VC investments. If a good business cycle cannot be built, the achievements of open source cannot be truly maintained. This is the author's usefulness.
Interesting reading, thanks for sharing. This is quick summary:
The articles compare the performance of different MQTT brokers implemented in different languages, including D, C, Erlang, Go, Java and C++. The basic conclusion should be D impl is the winner in the loadtest case, but C (Mosquitto) are the total winner for two case: loadtest + pingtest. C++ impl is not good. And the author said he "really don’t want to write C++ again unless I have to".
Basically, the conclusion of the article is nice. For MQTT brokers, the primary case sub-to-pub is just a layer-7 forwarding. This is an IO intensive scenario, if your MQTT parser is not written too badly.
So, for the impls done in system level languages, like C or D should archive the similar performance. Here, C++ impl should be an exception, although I haven't looked into the exact reason.
These blog articles, in fact, are quite old. The 100k message/second got with hundreds of connections is top ranked. However, the current Intel 12th Gen notebook can run at 250k mps with just 1 connection. I have tested (and read) several open-source MQTT clients and brokers. These MQTT brokers including impls done in C, Java and Erlang, can run at 1 million to 2.x million mps in a single modern Xeon socket but hard to go upward.
The truth here is that, the free lunch given by the efficient modern kernel TCP stack is just at 1-2M mps. If you want a faster broker, you need to make this kind layer-7 forwarding faster in the computational part, i.e. faster parsing and parallelizing. Traditional optimization tricks comes back again, like cache-friendly coding. The result is that we reached around 8 million mps in the same modern box even with message parsing, data routing and data dump all done in the first version of JoinBase(https://joinbase.io/).
The free lunch from the Moore's Law is truly over. We should be prepared for this.
Great. The OIDBS is using a built-in MQTT client. So, if you want o benchmark your server, just run the `import` command like as shown in the quick start video:
CURRENT_OIDBS_DIR is a hacky ENV var, for finding the models, should be removed in the future.
-n: is for the built-in model name, only the OIDBS repsect two models: nyct_lite and nyct_strip, you can get the two model sources from the article
-d: is for importing data only without injecting schemas. This is common if you are benching against a MQTT broker or server as you said. In fact, with this opt enabled, you can import any data source iff the data source is CSV or JSON. The benchmark logic is just to read and send the message line by line. That is, the files in the nyct_lite and nyct_strip directory is can be any CSV or JSON format. And the files'name is not important, but the model name still to be one of above two. Because the code checks the name.
(Ok, OIDBS may let the model name to be optional for such case, I will record this issue. thanks!)
If you have any problem, you can ask details in the community. I will help you as possible.
Great to see the this SBC benchmark. I have done one benchmark for several available SBCs from an edge data stack viewpoint (Edge Data Stack Capabilities Ranking: https://joinbase.io/ranking/, but the article is still in progress...so sorry for no content now)
What I have tested : $20 RockPi S(minimum squared Armv64 dev board), $0 unused IPTV box (S905L2 chip)(should be bought from secondhand market in ~ $7), #30 NanoPi Neo3(the minority of cheap SBC with DDR4 on), the Allwinner D1(the first batch avaiable Risc-v 64 board) and my own Xeon server (and more cloud based instances).
A possibly surprising fact is that, if the modern software (such as my database here) can take full advantage of the modern hardware, then the performance of even a small 40mm squared SBC board (such as RockPi S) can be compared to the large-scale-used traditional softwares running on a Xeon Platinum 8260 based bare-metal server(such as PostgreSQL).
There are SQLite(OLTP), DuckDB(OLAP) and some engine-based project like mentioned Apache Arrow(https://arrow.apache.org/)(OLAP): Apache Arrow has many language implementations, some do not include the query engine(for example, Rust implementation, which depends on the DataFusion for more SQL-like analytics) in its own repo, but other do include(for example, C++).
There is a comprehensive benchmark by ClickHouse for OLAP but including kinds of embedding engines: https://benchmark.clickhouse.com/
The more interesting is that, in fact, we have not an embedded HTAP engine. One of my database products already implements 3/4 HTAP at the engine layer, but unfortunately it's still just a free software, not an open source implementation.
I am the founder of this database product. A fundamental curiosity in my heart is, after tens years of development, can the bottom layer of the common-users-accessible database still develop?
JoinBase, is my initial answer to this question.
1. For the first time, it proposes the evolution of the database from the user's point of view.
2. For the first time, it well supports three DB payload types in a single engine of a free industrial-grade database: TP single row transcation write, TP and AP read.
3. For the first time, it is so fast that for most businesses, it is no longer necessary to use a distributed database for their data needs.
As the first end-to-end IoT Database, it is interesting to see how we fit IoT domain data storage and analysis much better than existed open source projects.
I recommend one ClickHouse compatible OLAP database project in Rust: [TensorBase](https://github.com/tensorbase/tensorbase/) for anyone who likes working with AP-side DBs on Rust.
FYI, recent information and progresses for TensorBase:
1. TensorBase(TB, for short) is not an reimplementing or clone of ClickHouse(CH, for short). TensorBase just supports the ClickHouse wire protocol in its server side.
2. TB's in-Rust CH compatible server side is faster than that in-C++ of CH. TB enables *F4* in the critical writing path: Copy-Free, Lock-Free, Async-Free, Dyn-Free (no dynamic object dispatching).
The result of TB's architectural performance: the untuned write throughput of TB is ~ 2x faster than that of CH in the Rust driver bench, or ~70% faster by using CH own ```clickHouse-client``` command. Use [this parallel script](https://github.com/tensorbase/tools/blob/main/import_csv_to_...) to try it yourself!
5. For complex groupby aggregations, recently we help to boost the speed of the TB engine to the same level of ClickHouse(not released, but coming soon).
It is often doubtful if one uses the word "fastest". You often see that one micro-bench lists ten products, then it says "look, I am running in the shortest time".
The problem is that, people often compare "apple to orange". Do you know how to correctly use ClickHouse(there are 20-30 engines in ClickHouse to use. Do you compare an in-memory engine to an disk-persistent-design Database?), Spark, Arrow... ? How can you guarantee to do a fair evaluation among ten or twelve products?
Remote: YES
Willing to relocate: YES
Technologies: Database, SQL, Rust, Java, JVM, Scala, WebAssembly, WASM, React, C/C++, Python, Javascript, Typescript, HTML5, CSS, TailwindCSS, Node, REST, Tauri, Linux, Kernel, IO_URING, BPF, LLVM, MLIR, LLM, LLAMA, ChatGPT, Finetuning, Vector Database, RAG, chatpdf, chatbi, chatexcel, BI, Analytics, Bigdata, Datalake, Data Warehouse, ClickHouse, PostgreSQL, Spark, Flink, Impala, Iceberg, Hudi, Kudu, Parquet, Arrow, SQL on hadoop, Redis, Distributed, AWS, GCP, Azure, SIMD, Lockfree, GPU, WebGPU, CUDA, ML, PyTorch, Triton, Compilation, JIT, VM, Docker, Cloud, MQTT, HTTP, TCP, IoT, Time series, Gateway, Ethereum, EVM, ZK
Résumé/CV: Send upon request
Email: jin.phd at gmail.com
14 years in work experience and 21 years in commercial software development, focused on engineering and science fields such as databases, compilers, deep learning, language design, high-performance computing, high-concurrency backend and network protocol and communication from data centers, to clouds and to decentralized internet.