Congrats on the release. Can dstack work with kubernetes? Like say another team owns the infrastructure and I’m required to use k8s, is it interoperable at all?
This seems like a good model for sustaining open source, but raises some questions.
Does anybody know how the DuckDB foundation works? The sponsors are MotherDuck, Voltron, and Posit, which are heavily venture-funded. Do DuckDB Lab employees work on sponsored projects for the foundation?
I am also curious if anyone can shed light on what kind of contract work DuckDB does to align its work with the open source project. This has always seemed like the holy grail, but it is difficult to do in practice.
Sounds like you are writing a pipeline to enrich a stream of data with historical data from Snowflake. This is a fairly common pattern. If the data in snowflake is not changing often, you would want to cache that somewhere for use with the stream processor to avoid that query overhead and speed things up.
Most of those listed can meet your first 2 requirements. Looking further down the list, your requirement of SQL and a DAG type of representation will limit the list to only a few. I don't know if many of those listed provide both of those capabilities.
If you relax the SQL constraint, more of them are applicable like Bytewax and Kafka-streams.
I recently read this article (https://materializedview.io/p/from-samza-to-flink-a-decade-o...) about Flink and it commented on Flink grew to fit all of these different use cases (applications, analytics and ETL) with disjoint requirements that Confluent built kafka-streams, ksql and connector for. What of those would you say Arroyo is better suited for?
Maybe this is a stupid question, but how would airbnb be looked at using this as precedent. If I sign up for airbnb and use their automatic price optimization engine would this fall under the same algorithmic collusion?
Great article! The breakdown of ETL, Analytical and Production workloads is so critical to understanding the pieces of the ecosystem and the history from your experience makes it much easier to understand.
I'm always curious about who writes all the comments on Hacker News. I am more of a lurker than a commenter.
We built this live Grafana Dashboard that uses Bytewax and Proton to pull data from the Hacker News API, process it as a stream, and then present it in a live updating dashboard.
Maybe this is a contradictory thought, but this is probably a good thing. I think you could argue that most drug mules are doing it because they don't, or don't think, they have another option to change their current life. If this reduces the number of humans used to smuggle drugs, it is probably a good thing.
These types of stories are what drove me to software in the first place. The ideal of a group of people working towards the same goal, for sometimes different reasons, but not for the sole purpose of monetary reward.
Good luck with the project!
I also echo some of the others, it would be great to see a sort of comparison with the alternatives so that we can better understand how the project fits in.
I worked at GitHub and this is exactly what I always wanted as someone working with data and ML artifacts. We even prototyped something with GitLFS once that looked like a very hacky and less well-thought-out version of this. Great work.
I am often confused on how things like imperative stream processing frameworks are supposed to work with the streaming database, if at all. Like what are you supposed to interact with the streaming database with? Is it only for streaming ETL? Do you need another application layer that can natively integrate, orchestrate and function. Maybe using a stream processing framework with a streaming database could facilitate a full application stack.
Thanks for sharing. I hadn’t heard of many scenarios in recent years, but I’m not sure if that’s just because nobody advertises that they had to put the source code in escrow.