Running a high-scale postgres in production is fraught. We have a really demanding workload and have hit a lot of undocumented performance cliffs. Most of these stem from fundamental design flaws in postgres [0][1].
I'm glad to see projects like this trying to using new tools to improve upon these deeply rooted issues that was just not practical even 12-18 months ago. This is really ambitious and I'm excited to see where this goes!
Founder here. As a backend developer, we often find ourselves designing beautifully encapsulated API’s for our app’s transactional features.
However when it comes to analytical data, we often leave that to data engineers to delve into the guts of our core db in order to ETL or virtualise the data for reporting.
And when we add/remove/change a column or table, we unknowingly break data pipelines or mess up reporting data downstream.
So I went about building a tool to expose analytical data through an explicit data contract, defined through a PostgreSQL schema, that can be documented, versioned and life-cycled like regular APIs but are still natural to consume by data engineers or analysts using SQL. It deploys as a sidecar alongside existing apps/databases so as to fit neatly within existing SDLC’s. The ultimate vision is that having multiple teams using Ansilo will naturally lend itself to a state where analytical data becomes an open ecosystem, similar to APIs, rather than the current tendency to end up as silos.
I thought the HN community may find it interesting, as it involved building a few somewhat novel components. To list a few:
- Generic framework for creating Postgres foreign data wrappers (FDW), supporting relational, NoSQL or file-based sources with advanced SQL push-down.
- Multi-protocol proxy for supporting native postgres protocol and HTTP connections over the same port.
- Zero-copy Rust-JDBC bridge over JNI.
Please let me know if you have any thoughts or questions. I’d love to hear any feedback!
I'm glad to see projects like this trying to using new tools to improve upon these deeply rooted issues that was just not practical even 12-18 months ago. This is really ambitious and I'm excited to see where this goes!
[0] https://www.recall.ai/blog/postgres-postmaster-does-not-scal...
[1] https://www.recall.ai/blog/postgres-listen-notify-does-not-s...