We're Building Postgres in Rust. Using the LLVM of Databases(turso.tech)
turso.tech
We're Building Postgres in Rust. Using the LLVM of Databases
https://turso.tech/blog/a-new-modern-version-of-postgres-in-rust
10 comments
People said same thing about sqlite, but I love turso sqlite and haven't faced any issues.
There is no turso sqlite
I don't know if you were being pedantic or confused, but turso is a sqlite3 remake in rust (with some additional features).
turso is not an sqlite remake.
at all
at all
"An in-process SQL database, compatible with SQLite, and now with a Postgres frontend too "
I'm already quite excited about Turso being SQLite-compatible, but adding many features on top.
And when a feature is not directly compatible with SQLite (ie: you can't directly read the file with `sqlite3`, it's straightforward to convert). This is great because you know you'll always be able to continue working with that database. Even if Turso stopped working, it's still a valid SQLite database.
A combination I would be excited about is:
- Full support for Postgres protocol/wire format (ie: Postgres, but in-process, backed by a single file). - Optional: Client/server architecture for further scaling and remote management using existing Postgres tooling - All backed by a SQLite-compatible file
They are already adding MVCC to SQLite anyway. So their effort seems doable, and I hope they succeed.
And when a feature is not directly compatible with SQLite (ie: you can't directly read the file with `sqlite3`, it's straightforward to convert). This is great because you know you'll always be able to continue working with that database. Even if Turso stopped working, it's still a valid SQLite database.
A combination I would be excited about is:
- Full support for Postgres protocol/wire format (ie: Postgres, but in-process, backed by a single file). - Optional: Client/server architecture for further scaling and remote management using existing Postgres tooling - All backed by a SQLite-compatible file
They are already adding MVCC to SQLite anyway. So their effort seems doable, and I hope they succeed.
Supporting Postgres is a good goal but honestly the real challenge is extensions. Would supporting Postgres wire compatibility guarantee any Postgres extension would also work? This is one of the problems with Aurora - it’s all well and good until you need an extension that isn’t on the blessed list
There’s a section about extensions in the FAQ.
> Would supporting Postgres wire compatibility guarantee any Postgres extension would also work?
It would not, for obvious reasons.
It would not, for obvious reasons.
Care to elaborate? The reasons are not obvious
Wire protocol applies to external interaction with the db.
Internal ABI/API is used by extensions to directly interact with core subsystems and depends on internal models for things like storage.
It might be analogous to HTTP versus an nginx plugin.
Internal ABI/API is used by extensions to directly interact with core subsystems and depends on internal models for things like storage.
It might be analogous to HTTP versus an nginx plugin.
curious how hard it is to implement wire protocol compatibility vs the internal api surface. Are we talking like, an order of magnitude more work here? I wonder if, for example, Aurora team are literally maintaining forks of extensions for their implementation and that's why only a subset of all possible extensions are supported?
Sounds cool but also why? Is it so it's possible to have 1 database and then switch from sqlite to postgres without data migration?
The llvm of databases is not turso, it's datafusion.
datafusion is llvmm for etl engines, they don't have indexes or transactions for example.
Yes I assumed datafusion when I heard that phrase.
> If you have ever babysat a REFRESH MATERIALIZED VIEW cron job, or faked live views with a pile of triggers, read that sentence again: our views update themselves, live. This is the feature Postgres users have quietly wanted for twenty years.
Real-time materialized views - so far mostly Chinese tech shops have solved this - look at Apache Doris, StarRocks - but these are built on top of MySQL then maybe RisingWave which is Postgres wire compatible.
having real-time materialized views means you can take out 1 or 2 infra pieces from your stack e.g Flink & Kafka - for some simplified use cases. that means no more ETL jobs for some use cases.
Real-time materialized views - so far mostly Chinese tech shops have solved this - look at Apache Doris, StarRocks - but these are built on top of MySQL then maybe RisingWave which is Postgres wire compatible.
having real-time materialized views means you can take out 1 or 2 infra pieces from your stack e.g Flink & Kafka - for some simplified use cases. that means no more ETL jobs for some use cases.
Would be interested in strategic value add versus cedardb (although cedar is closed source in a way that at the moment I think may kill it)
As someone who doesn't follow databases that closely, how is this different from Amazon Aurora (other than being open source)?
Will it have fast COUNT-s?
[0]: https://x.com/glcst/status/2077759127682486561?s=46