Stream Processing with DuckDB/Polars?
2 comments
Two ideas you might like:
Debezium + Arrow Flight: Use Debezium as a library to grab PostgreSQL CDC events and stream them into Arrow for super-fast, columnar processing. Works great with Polars or DuckDB.
RisingWave: This is a lightweight stream processor that connects directly to Postgres CDC, lets you write SQL for transformations, and keeps everything updated in real-time. No Kafka or heavy setups required.
Debezium + Arrow Flight: Use Debezium as a library to grab PostgreSQL CDC events and stream them into Arrow for super-fast, columnar processing. Works great with Polars or DuckDB.
RisingWave: This is a lightweight stream processor that connects directly to Postgres CDC, lets you write SQL for transformations, and keeps everything updated in real-time. No Kafka or heavy setups required.
Have you seen pgstream? https://github.com/xataio/pgstream
It is similar to pg_replicate and could be a good fit for streaming CDC data from Postgres. There is no built-in output plugin specifically for DuckDB but it might help you for building something lightweight and custom for your use case.
Thanks, I'll take a look!
Doing some research a little bit surprised there isn't too much out there in terms of trying to solve for these relatively simple single-node processing situations. There are some things (e.g. pg_replicate) but would like something that's more oriented around the Arrow data ecosystem. Curious if anyone has either managed to build anything custom here that worked well, or any tools I'm missing.