Sorry, I didn't see this question when it came in (the comment was collapsed in the thread for some weird reason).
The wikidata image I was using (sourced from a huggingface dataset Arun Sharma kindly uploaded for LadybugDB) is about 10GB zipped I think, and when imported into Slater takes up about 20GB of disk space. It's about 133GB of raw Cypher.
So in bytes/edge, that's probably about 14.
In terms of how to shard (which I'm taking to mean allocation of nodes to ISAM blocks) during an initial build or during a consolidation step, it will try to pack neighbours into the same blocks along with metadata about neighbouring segments, with some additional heuristics to avoid superhubs causing over-filling issues. For on the fly writes it just uses a normal WAL that it builds tables out of until theres a segment merge, at which point it then uses the same logic as for the consolidation.
In terms of canonicalising, it asks for a pk property during import, and then uses a (label, key property, value) tuple for uniqueness. So in a sense it doesn't really do that itself, it relies on that being handled outside the import.
Hope that helps.
PS as a follow-up: one really important memory-saving trick I used was Elias-Fano encoding on the adjacency matrices. Each node keeps an array of nodes it's joined to, and Elias-Fano does an amazing job of compression on those, both on disk and in RAM.
Not a weakness so much I guess, just a different cost/performance trade-off choice that I think largely was a direct result of having been written in java for enterprise purposes at a time when Rust didn't yet exist. "Low memory" and "written in Java" are at least in my experience not terribly compatible goals.
Ha - ok understood, although in my specific case I think this is better than anything I would have written. I edited it pretty hard to get it to say what I wanted it to say, so not sure if that counts. Thanks anyway for the feedback.
Hi all - I just wanted to announce a low memory footprint FOSS graph db I've been working on (named Slater, after the Archer character). The most common complaint about Graph/GraphRAG DBs is the cost and memory footprint they consume: many depend on holding the whole dataset fully in RAM, which makes them expensive to run.
Slater starts with the premise of a fixed memory budget applied to an LRU cache of what's on disk, then uses ISAM blocks and DiskANN/Vamana/PQ to allow paging the contents of the graphs and vectors you need into that cache. It's designed around read-heavy-write-light cases, and can be backed either by local disk or by S3/GCS buckets with an optional sized local disk L2 cache as well.
Speaks standard Bolt, and is GDPR friendly (encryption at-rest and in-transit). Multi-user-multi-graph with ACLs, Rust-with-forbid-unsafe and NFS-friendly too (no mmaps). It's Apache licensed.
Please give it a try if you get a chance. Would love any suggestions or feedback.
Full disclosure: yes, it was authored by Claude Code, although I provided the storage model and design it used, along with code samples and influences from other open source projects like FalkorDB and Memgraph for Bolt wire-compatibility.
The wikidata image I was using (sourced from a huggingface dataset Arun Sharma kindly uploaded for LadybugDB) is about 10GB zipped I think, and when imported into Slater takes up about 20GB of disk space. It's about 133GB of raw Cypher.
So in bytes/edge, that's probably about 14.
In terms of how to shard (which I'm taking to mean allocation of nodes to ISAM blocks) during an initial build or during a consolidation step, it will try to pack neighbours into the same blocks along with metadata about neighbouring segments, with some additional heuristics to avoid superhubs causing over-filling issues. For on the fly writes it just uses a normal WAL that it builds tables out of until theres a segment merge, at which point it then uses the same logic as for the consolidation.
In terms of canonicalising, it asks for a pk property during import, and then uses a (label, key property, value) tuple for uniqueness. So in a sense it doesn't really do that itself, it relies on that being handled outside the import.
Hope that helps.
PS as a follow-up: one really important memory-saving trick I used was Elias-Fano encoding on the adjacency matrices. Each node keeps an array of nodes it's joined to, and Elias-Fano does an amazing job of compression on those, both on disk and in RAM.