- Materialize
- Flink SQL
- Arroyo
- Readyset
- RisingWave
- Timeplus
- Pathway
- Dozer
- ReadySet
- Snowflake dynamic tables
- Native materialized views in OLTP databases
- Just having a stack of views in your db
- Poor man's MVs with triggers
All subtly different on every spectrum from consistency, UDF support, operator support, latency, scaling/state limits, source/sink integrations, and compatibility with existing protocols.
What seems unique is the focus on "writebacks to the source without Kafka/Connect in between", instead of having either a built-in cache, serving as a stream processor, or both. It looks like the built-in cache is still available through the FDW deployment pattern.
They note that relative to the source tables they are eventually consistent (of course, unless you want to delay transaction writes) but it's not clear what other consistency aspects they respect (such as preserving transactions end to end).
Overall this looks like it's designed to overcome materialized view limitations (which in popular OLTP dbs are pretty severe w.r.t. either what operations are supported, latency, or both) compared to other solutions that basically move the action downstream...curious if it will see much use, or if they'll inevitably introduce sinks and direct access to see if they can compete in the "live ODS" segment with Materialize and RisingWave.
edit: to make my comment more clear: this is a new entrant in a crowded space with several sophisticated, established players and the main differentiation is the deployment pattern. I'd be curious to know if anything else sets them apart
Agree with this. Snowflake has best-in-class dev experience and performance for Spark-like workloads (so ETL or unconstrained analytics queries).
It has close to worst-in-class performance as a serving layer.
If you're creating an environment to serve analysts and cached BI tools, you'll have a great time.
If you're trying to drive anything from Snowflake where you care about operations measured in ms or single digit seconds, you'll have a bad time and probably set a lot of money on fire in the process.
If you want an Airflow-ish approach without punishing your future self, pick Prefect. Otherwise go with Temporal. Above all do not adopt Airflow for the use cases you describe in 2023
"Lakehouse" usually means a data lake (bunch of files in object storage with some arbitrary structure) that has an open source "table format" making it act like a database. E.g. using Iceberg or Delta Lake to handle deletes, transactions, concurrency control on top of parquet (the "file format").
The advantage is that various query engines will make it quack like a database, but you have a completely open interop layer that will let any combination of query engines (or just SDKs that implement the table format, or whatever) coexist. And in addition, you can feel good about "owning" your data and not being overtly locked in to Snowflake or Databricks.
There is a huge round of "data observability" startups that address exactly this. As a category it was overfunded prior to the VC squeeze. Some of them are actually good.
They all have various strengths and weaknesses with respect to anomaly detection, schema change alerts, rules-based approaches, sampled diffs on PRs, incident management, tracking lineage for impact analysis, and providing usage/performance monitoring.
Datafold, Metaplane, Validio, Monte Carlo, Bigeye
Great Expectations has always been an open source standby as well and is being turned into a product.
I think it's more a matter of comparing minivans (cloud "DWH" engines) to sports cars (Clickhouse et al) here.
Snowflake's performance characteristics & ops paradigm have always been more consistent with managed Spark than anything else. Thus the competition with Databricks. They have only recently started pretending to be anything than a low-maintenance batch processor with a nice managed storage abstraction, and their pricing model reinforces this.
That being said, for now it's pretty hard currently to find something that gives you:
- Bottomless storage
- Always "OK" performance
- Complete consistency without surprises (synchronous updates, cross table transactions, snapshot isolation)
- The ability to happily chew through any size join and always return results
- Complete workload isolation
...all in one place, so people will probably be buying Snowflake credits for a few years yet.
I'm excited about the coming generation--c.f. StarRocks and the Clickhouse roadmap--but the workloads and query patterns for OLAP and DWH only overlap due to marketing and the "I have a hammer" effect.
I don't think the slight misuse of either type of engine is bad at small-to-medium scale, either. It's healthy to make "get it done" stacks with fewer query engines, fewer integration points, and already-known system limitations.
I'm a heavy Prefect user and was also very confused about the initial rewrite, even after reading several summaries. My best advice is to just try using 2.0 (Orion). Here's how I'd summarize the difference:
Prefect 1.0 feels like second-gen Airflow--less boilerplate, easy dynamic DAGs, better execution defaults, great local dev, etc etc. It's more sane but you still feel the impedance mismatch from working with an orchestrator.
Prefect 2.0 is a first-principles rewrite that removes most of the friction from interacting with an orchestrator in the first place. Finally, your code can breathe.
+1, the Pandas API is somewhere between mediocre and bad, and results in garbage code unless you use it in a carefully constrained way (which is admittedly true of many complete languages, much less libraries that organically evolved several tooling generations ago)
It's worth noting that hydraulic fracturing itself is rarely the problem. Issues come from moving fluid volumes somewhere else during production: subsidence due to extraction from the place where the hydrocarbons are, or (most often) from injecting produced water into disposal wells, eventually triggering faults. Some of that produced water was introduced by the operations, but most of it was just in the ground with the oil & gas being produced.
It's possible to directly trigger small faults with while fracturing the rock, or to do something stupid like fracture into a freshwater zone other people are using, but that's not what's driving quakes in e.g. Oklahoma.
+1, Migadu is simple, reliable, easy, and just works. Cost is low enough it might as well be free. I've only ever had one issue (from incorrectly interpreting documentation) and had an actual human respond and walk me through it.
The only downside to be aware of is the lack of calendar support (technically yes via CalDAV, but that doesn't work for most users--e.g. Calendly won't work).
Edit: the way they handle domains and email aliases has simplified my email life.
In that situation (dual usage modes) I think I'd rather have the primary data store be Materialize, and just snapshot Materialize views back to your warehouse (or even just to an object store).
Then you could use that static store for exploration/fixed analysis or even initial development of dbt models for the Materialize layer, using the Snowflake or Spark connectors at first. When something's ready for production use, migrate it to your Materialize dbt project.
The way dbt currently works with backend switching (and the divergence of SQL dialects with respect to things like date functions and unstructured data), maintaining the batch and streaming layers side by side in dbt would be less wasteful than the current paradigm of completely separate tooling, but still a big source of overhead and synchronization errors.
If the community comes up with a good narrative for CI/CD and data testing in flight with the above, I don't think I'd even hesitate to pull the trigger on a migration. The best part is half of your potential customers already have their business logic in dbt.
Anyone here with practical experience using [Flying Squid](https://github.com/HazyResearch/flyingsquid) over the open-source Snorkel library? I'm curious if this platform re-uses some of that line of research or if it's not practical for some reason.
- Materialize - Flink SQL - Arroyo - Readyset - RisingWave - Timeplus - Pathway - Dozer - ReadySet - Snowflake dynamic tables - Native materialized views in OLTP databases - Just having a stack of views in your db - Poor man's MVs with triggers
All subtly different on every spectrum from consistency, UDF support, operator support, latency, scaling/state limits, source/sink integrations, and compatibility with existing protocols.
What seems unique is the focus on "writebacks to the source without Kafka/Connect in between", instead of having either a built-in cache, serving as a stream processor, or both. It looks like the built-in cache is still available through the FDW deployment pattern.
They note that relative to the source tables they are eventually consistent (of course, unless you want to delay transaction writes) but it's not clear what other consistency aspects they respect (such as preserving transactions end to end).
Overall this looks like it's designed to overcome materialized view limitations (which in popular OLTP dbs are pretty severe w.r.t. either what operations are supported, latency, or both) compared to other solutions that basically move the action downstream...curious if it will see much use, or if they'll inevitably introduce sinks and direct access to see if they can compete in the "live ODS" segment with Materialize and RisingWave.
edit: to make my comment more clear: this is a new entrant in a crowded space with several sophisticated, established players and the main differentiation is the deployment pattern. I'd be curious to know if anything else sets them apart