I definitely get your point, but over-generalized comments like these are also dangerous.
Just as there are many MBAs who were or are veteran software developers, the HN community is large enough that there are many members who are professional investors.
Sorry if it's a bit of an enterprisy-response, but please reach out! We are supporting some non-cloud commercial customers on a case-by-case basis. The reason for this is that we find the support and maintenance burden to be much higher with a non-cloud delivery model, which isn't always a great experience for either party. We also have a managed product (where the data plane resides in your environment) that may work, depending on your infra and security requirements.
Re: custom code -- our codebase is fully source-available and open to contributions, but the source+sink code going through some refactoring to make it more beginner-friendly. Depending on your consistency requirements, we also support Debezium and our own CDC format (https://materialize.com/docs/connect/materialize-cdc/) for folks who want to bring in their own data sources. (For quick prototypes, we also support csv/json/plaintext source types, as well as SQL INSERTs!)
Exactly! Since most ELT tools (including Airbyte) support json + csv output formats, those work perfectly well with Materialize out-of-the-box. I'm playing around with Slack+Stripe Airbyte sources to try and come up with some fun dashboards to show off in Materialize as we speak.
Nothing we can share publicly at the moment yet, but if you reach out and chat, we're more than happy to give you some numbers that I think will address what you're looking for!
Do you happen to have any examples of real-time queries or apps you would be interested in?
Re: the second point — you’re right, Materialize has historically leveraged existing upstream systems (like Kafka) for things like persistence. But we also hear you loud and clear that not everyone wants to stand up Kafka :)
We’re very aware of ksqlDB. I would recommend this video from last week where Matthias talks about some of the strengths and weaknesses of ksql: https://youtu.be/KUQuegJ4do8
As George points out above, we haven’t added our native persistence layer yet. Consistency guarantees are something we care a lot, so for many scenarios, we leverage the upstream datastore (often Kafka).
But to answer your question, yes, our intention is to support separate cloud-native storage layers.
Materialize is a bit more of a lower-level technology, and less of a single "feature". I've included some example use cases built on top of Materialize in my other comment.
In addition, while Materialize does support connecting to other databases, to power "real-time materialized views", a common architecture we are used for is to present a SQL view on top of streaming systems (such as Kafka or Kinesis).
Here's an overview of Materialize that explains the relationship between Timely Dataflow, Differential Dataflow, and Materialize, starting at 23:20 -
https://materialize.io/blog-cmudb/
(Responding from the perspective of Materialize, the technology and company built on top of Differential Dataflow) Some examples we're seeing interest for:
Building realtime, low-latency (< 10 sec) dashboards. The type of things where you previously would have had to wait for several hours or a day for ETL pipelines to crunch through a lot of numbers.
We're also fielding interest for streaming ML applications. Ie, moving from batch models to streaming models.
Also worth pointing out that Differential Dataflow has been around for awhile, while Materialize is fairly young. We're still constantly learning about new applications!
I'm a PM @ Materialize. Regarding the questions from the OP's link:
#1 -- Postgres is built in C, while Timely/Differential (which underpin Materialize) are built in Rust. Materialize could be a Postgres extension at some point, but for now we want to control the stack (to more easily do cross-dataflow optimization, etc)
#2 -- We are absolutely interested in exposing an API to enable new data sources (and data sinks!). It currently takes a bit of time to understand some basic Timely Dataflow concepts, but we intend on documenting this and opening things up. We're also trying to understand user requirements around things like durability guarantees and UPDATE/DELETEs (Feel free to email me or hit us up in our public Slack as well).