I'm working on an evolutionary, accumulate-only database system called SirixDB written in Java (and a module in Kotlin) in my spare time and in my day to day job I'm also working as a software engineer (embedded; besides I'm dancing a lot -- swing dances, mainly Lindy Hop and travel to festivals in Europe often times in my holidays) :-)
Furthermore, I'm working on Brackit (http://brackit.io), a query engine for JSON and XML using a JSONiq like query language. It can be used as ad-hoc in-memory query processor as the query processor of a data store. We envision, that it encapsulates common proven optimizations whereas the individual data stores can add further stages to rewrite the queries for index matching and physical optimizations.
SirixDB (https://sirix.io or https://github.com/sirixdb/sirix) is all about efficient versioning of your data. That is on the one hand it reduces the storage cost of storing a new revision during each transaction-commit while balancing read- and write-performance through a novel sliding snapshot algorithm and dynamic page compression. On the other hand Sirix supports easy query capabilities for instance to open a specific revision by a timestamp or several revisions by a given timespan, to navigate to future or past versions of nodes in the tree-structure and so on. It basically never overwrites data and is heavily inspired by ZFS and Git and borrows some ideas and puts these to test on the sub-file level.
In stark contrast to other approaches SirixDB combines copy-on-write semantics with node-level versioning and does not require a write-ahead-log for consistency.
It all started around 2006 as a university / Ph.D. project of Marc Kramis and I worked on the project since 2007 and already did my Bachelor's Thesis, Master's Thesis as well as several HiWi-Jobs on topics regarding the project and I'm still more eager than ever to put forth the idea of a versioned, analytics plattform to perform analytical tasks based on current as well as the history of the data.
It's first and foremost a document store / so for JSON or XML currently, that's why it's not SQL compliant. I think other query languages as JSONiq are much more tailored to this, albeit a niche of course (based on XQuery).
Regarding tamper proof audit logs not much is missing. Cryptographic hashes instead of XXH3, a commit hash chain and signed commits. Actually, I think that's a great addition with minimal changes needed.
What you can audit currently is "who changed what" for instance.
Besides, documents are split into fine granular nodes, so we have "no" upper limit besides running out of 48bit nodeKeys/node identifiers at some point maybe.
Furthermore, a (versioned, as always) path summary keeps track of all distinct paths, which is a key ingredient for the optional secondary indexes, which can index paths or paths and content among other stuff as simply indexing fields. Optionally you can also add indexes to speed up aggregate queries (which are basically column projections).
Furthermore, the whole storage can self-validate through checksums stored in parent pages up to the root as in ZFS for instance.
DeweyIDs can be optionally stored for each node, which can speed up the comparison of subtrees between revisions (in order to detect simply which changesets belong to a certain subtree without having to traverse ancestor chains). They lend themselves well for compression.
We even have importers which can identify based on heuristics minimal edit operations to import existing revisions of XML or JSON documents and to commit these with hopefully a minimal or close to minimal set of update operations, but it depends heavily on the data.
I'm working on a side-project in my spare time since the end of 2012 (before at the university) with some gaps, which is an append-only DBS with time travel capabilities using a custom storage engine based on COW tries and a (still I think) novel page versioning strategy called Sliding Snapshot. Recently I began work on pooling pages for reuse and a custom allocator for variable page sizes. Before, I had created new instances whenever a page was read from disk instead of reusing instances, so the allocation rate was really high for parallel transactions.
I once implemented the backend of a calendar and resource control for a low code platform.
The control is highly customizable, with a lot of views to chose from, daily, monthly, yearly... but also resource views (you can book resources with custom groupings, by plugin, by the resource-ID, whatever...), define "plugins" on the data sources, what's the from- and to- columns, the title column, what's the resource (may be from a foreign key / 1:1 relationship or 1:N if it's from a "child" data source or from the same data source/table).
Furthermore I've implemented different appointment series, to chose from (monthly, weekly (which weekdays), daily...), which column values should be copied. Also appointment conflicts (or only conflicts if they book the same resource). You could also configure buffers before and after appointments where no other appointment can be.
That was a lot of fun and also challenge sometimes regarding time zones and summer/winter time in Europe and so on :-)
So, I think in my local bubble noone is for instance as excited about DB systems as I am, so in essence I thought I could even spend some money to get some expert opinions or rather insights I'm struggling with (currently for instance with bad throughput of my immutable OSS DBS). That said I think noone so far wanted money and some even offered help, but so far I think they didn't have time, thus didn't answer any "pings". So, as I can't spend too much time (and of course not too much money) either on profiling and debugging right now it's kind of a dilemma, as it would IMHO be very interesting to know what's slowing down N read-only trxs in my system :-) that said a couple of years ago I also asked about help with a frontend without much luck. I guess it has to have some value of course, so maybe at least spending some money (even if it's a non profit spare time project since 11 or even more years) should be OK :-)
Throughput. The code can be "suspended" on a blocking call (I/O, where the platform thread usually is wasted, as the CPU has nothing to do during this time). So, the platform thread can do other work in the meantime.
We're using a similar trie structure as the main document (node) index in SirixDB[1]. Lately, I got some inspiration for different page-sizes based on the ART and HAMT basically for the rightmost inner pages (as the node-IDs are generated by a simple sequence generator and thus also all inner pages (we call them IndirectPage) except for the rightmost are fully occupied (the tree height is adapted dynamically depending on the size of the stored data. Currently, always 1024 references are stored to indirect child pages, but I'll experiment with smaller sized, as the inner nodes are simply copied for each new revision, whereas the leaf pages storing the actual data are versioned themselfes with a novel sliding snapshot algorithm.
You can simply compute from a unique nodeId each data is assigned (64bit) the page and reference to traverse on each level in the trie through some bit shifting.
I think it depends, but I wonder if anything can be done about the problem with checked exceptions in lambdas / for instance the streams. I think the enhanced switch with handling failure is only part of the solution, but I'm also a proponent of having only unchecked exceptions.
What about storing the data and thus, the indexes in Kafka. Would it make sense? Let's say currently, I'm storing SirixDB resources in files. However, instead of offsets into a file the index pages could be stored in Kafka optionally (or Pulsar...). Is Kafka too slow for this or only for specific row-tuples? We could make a combined storage caching the pages locally or also storing in the file system and asynchronous storing in Kafk, S3 or whatever.
I'm working on an evolutionary, accumulate-only database system called SirixDB written in Java (and a module in Kotlin) in my spare time and in my day to day job I'm also working as a software engineer (embedded; besides I'm dancing a lot -- swing dances, mainly Lindy Hop and travel to festivals in Europe often times in my holidays) :-)
Furthermore, I'm working on Brackit (http://brackit.io), a query engine for JSON and XML using a JSONiq like query language. It can be used as ad-hoc in-memory query processor as the query processor of a data store. We envision, that it encapsulates common proven optimizations whereas the individual data stores can add further stages to rewrite the queries for index matching and physical optimizations.
SirixDB (https://sirix.io or https://github.com/sirixdb/sirix) is all about efficient versioning of your data. That is on the one hand it reduces the storage cost of storing a new revision during each transaction-commit while balancing read- and write-performance through a novel sliding snapshot algorithm and dynamic page compression. On the other hand Sirix supports easy query capabilities for instance to open a specific revision by a timestamp or several revisions by a given timespan, to navigate to future or past versions of nodes in the tree-structure and so on. It basically never overwrites data and is heavily inspired by ZFS and Git and borrows some ideas and puts these to test on the sub-file level.
In stark contrast to other approaches SirixDB combines copy-on-write semantics with node-level versioning and does not require a write-ahead-log for consistency.
It all started around 2006 as a university / Ph.D. project of Marc Kramis and I worked on the project since 2007 and already did my Bachelor's Thesis, Master's Thesis as well as several HiWi-Jobs on topics regarding the project and I'm still more eager than ever to put forth the idea of a versioned, analytics plattform to perform analytical tasks based on current as well as the history of the data.
e-mail address: [email protected]