Instrumenting would have been 100% just as feasible. Ironically it would have been more work.
For context, our DB is highly optimised for ingestion (millions of rows / second) and adding any high resolution metrics there would impact performance, so would have to be either ripped out afterwards or engineered very carefully (read "not cheap") as to not impact performance.
This stuff took an afternoon, is reusable, and frankly, was more fun to implement :-).
I suspect there are tools out there that that do this stuff, of course. The matter is still finding and learning how to use them compared to writing a few hundred lines of code.
Maybe we should write a blog post, though there's gotta be one out there for this already.
The short of it is: Learn C. Learn your system calls. Learn JNI. Learn about com.sun.misc.Unsafe. Learn about the disruptor pattern. Learn how to pool objects. The long type can pack a lot of data. Go from there!
I'd say both Rust and C++ trade blows when it comes to expressiveness. You know Rust already, so I'm not going to try sell you how powerful macros can be (see SQLx's ability to compile-time check SQL queries).
And indeed C++ templates are a lot more like Rust macros than Rust generics: They're turing-complete.
Joint with some interesting language choices like SFINAE (substitution failure is not an error), you end up with the ability to specialize functions, methods and whole classes in C++.
You can also have functions that return different types.
C++ templates work like duck-typing within a static language: In Rust you need to say what traits your generics need to support. In C++ it will try and substitute and if it fails (say, because T doesn't support the required methods) it will try another substitution until none are left.
If none of the substitutions work, you will be shown ALL of them in error reporting: This is what leads to pages and pages of compile errors of single-character typos in C++.
Templates are really cool, but also pretty confusing when reading code since you're in a guessing game of what types will fit the constraints imposed by the _implementation_ of the function.
From C++20 there's concepts to make templates work a little smoother.
There's been whole books written about how to abuse templates: these are pre-requisite knowledge when working in large codebases.
We have an initialisation step as soon as we load the jni lib that takes care of this. Given that this gets done before any other threads are started, I don't think there'd be an issue. Good point :-)
I'd say expressiveness of C++ with productivity of Java.
Rust is indeed not easy to learn.
A little example. Yesterday I was making changes to our C++ client library and I wanted to improve the example in our documentation.
We use a dedicated protocol called ILP for streaming data ingestion and each of the inserted rows has a designated timestamp.
In the Rust example, I using added support for chrono::DateTime and it was trivially easy for me to add a timestamp for a specific example date and time: Utc.with_ymd_and_hms(1997, 7, 4, 4, 56, 55).
Our C++ library instead takes an std::chrono::time_point. I wanted to use the same datetime. As far as I can tell it requires first going through the old C "struct tm" type (which is local and not UTC), then converting to "time_t" then converting to utc via gmtime and then constructing a time_point from that.
After 10 minutes the code got too long and complicated so I just substituted it a timestamp specified as an int64_t in nanoseconds.
Don't get me wrong, the C++ time_point is a work of art in how flexible it is, but unnecessarily complicated in most cases.
I should add that I also spent 45 minutes yesterday debugging a CMake issue.
Rust is not easy to learn, but it's just more modern and productive.
C++ is still great if you've got a massive team, but at our scale I don't think it makes any sense.
When the time is right. There's finally new APIs coming in the Java space that will make native-code interop easier and more reliable.
Our open source database edition can also be used embedded though, so we can only upgrade at the pace of our customers and because of that we still are compatible all the way down to Java 8.
Were it not for this detail, we'd probably consider it a lot sooner.
The focus of the article really is about JNI in Rust.
I see most questions are about "Why did you not use X language instead?", so let me try and address this.
To answer the "Why not just Rust", I should first mention that Rust was still in its early days (before 1.0), and it was a risky bet to choose an emerging language.
The project was started by Vlad (our CEO) who had a background writing high performance Java in the trading space. The Zero-GC techniques - whilst uncommon in open source software - are mature and a staple of writing high performance code in the financial industry. The product evolved organically, feature after feature.
I personally joined the team from a C and C++ background, having previously moved from a project that suffered from minute-long compile times from single .cpp files due to template overuse. Whilst I do miss how expressive high-level C++ can be, Java has really good tooling support.
When writing systems-style software most of what matters in terms of performance is how we call system calls, manage memory and debug and profile. This is an area where Java really shines. Don't get me wrong: In the absolute sense I think C++ tools tend to be better (Linux Perf is awesome!), but Java tooling is _there_. IntelliJ makes it trivially easy to run a test under the debugger reliably and consistently. It's equally easy to run a profiler and to get code coverage. The same tools work across all platforms too, might I add. It's not necessarily better, but it's easier. Turns out that while a little quaint, using Java turned out to be a pretty good choice in my opinion in practice.
Times have moved on. The Rust community really cares about tooling, and it's one of the reasons why we've picked it over expanding our existing C++ codebase: We just want to get stuff done and have enough time left in our dev cycle to properly debug and profile our code.
All the library does is facilitate the build process.
I think there's possibly some automation that can be added once that whole stuff stabilizes properly. From Rust it's already possibile to automate exposing a C header via cbindgen, so it wouldn't be too hard.
JNI ultimately still provides the most complete set of capabilities (e.g. calling back into Java).
For the rust-maven-plugin, we've accepted a PR to build binaries and we'll continue accepting PRs for any future enhancements such as the one you're mentioning.
Here at QuestDB we're beginning to write part of our database code in Rust as JNI native code extensions. Since all of our Java code is built with Maven we found we needed an easier way to invoke `cargo` during our normal build cycle. The plugin makes it easier and has a few extra features such as running tests.
If you also have a Java code base you're thiking of extending with Rust, we hope you can find this project useful.
Hi, I'm the original author of the QuestDB Python client library and benchmark.
It all started when we had one of our users needing to insert quite a bit of data into our database quickly from Pandas. They had a dataframe that took 25 minutes to serialize row-by-row iterating through the dataframe. The culprit was .iterrows(). Now it's a handful of seconds.
This took a few iterations: At first I thought this could all be handled by Python buffer protocol, but that turned out to create a whole bunch of copies, so for a number of dtypes the code now uses Arrow when it's zero-copy.
The main code is in Cython (and the fact that one can inspect the generated C is pretty neat) with supporting code in Rust. The main serialization logic is in Rust and it's in a separate repo: https://github.com/questdb/c-questdb-client/tree/main/questd....
We actually do have a web based demo which is https://demo.questdb.io/ preloaded with millions of rows of data.
That one focuses on SQL queries though.
The notebook in https://play.questdb.io/ offers a more rounded experience to try out any and all of our features.
You can use the notebook to try out data ingestion, dropping partitions and more that is simply not possible in a more sandboxed environment.
The other part is that we value our Python users and wanted to provide an example of how to use our database in conjunction with other tools commonly used in the data science space to slice and dice time series data.
Yes. We're pretty good with large volumes of data.
Eventually all local drives fill up though.
When ingesting data we partition data by time. By default we partition by day. This give you the flexibility to detach partitions, store them somewhere slower and cheaper with more capacity for longer term storage and reattach them later if need be.
Built on top of our open source primary product, we also have a cloud variant of QuestDB which runs on AWS. One of the things that we're building there is cold storage. It will automate this process onto S3 such that if a query ever needs to access this older data it will re-enstate it automatically for you with no admin overhead.
Most databases store the latest state of something.
We don't. We ingest events.
After all, life is a function of time :-)
The whole world ticks and we take those ticks and store them.
If part of your application tracks anything happening over time (trades, ocean pollution levels, ships moving, rocket simulation metrics.. or whatever else then it makes sense to store those events in a time series database.
What we provide, primarily, is two basic pieces of functionality:
(1) Taking in lots of events FAST. Our ingestion rate is high (and we also integrate with things like Kafka, Pandas -- see the notebook, etc). Each of our time series tables (we support regular ones too) comes with a special timestamp column.
(2) Specialized SQL to make sense of data that's changed over time, such as grouping and resampling by time and more. Take a look at our docs for things like SAMPLE BY, LATEST ON, ASOF JOIN, LT JOIN and more. On disk, we also guarantee that all records are sorted by time and this gives us great query performance for these time-based types of queries.
Instrumenting would have been 100% just as feasible. Ironically it would have been more work.
For context, our DB is highly optimised for ingestion (millions of rows / second) and adding any high resolution metrics there would impact performance, so would have to be either ripped out afterwards or engineered very carefully (read "not cheap") as to not impact performance.
This stuff took an afternoon, is reusable, and frankly, was more fun to implement :-).
I suspect there are tools out there that that do this stuff, of course. The matter is still finding and learning how to use them compared to writing a few hundred lines of code.