It is not obvious from the post but it seems like the allow list for the scripts supports whitelisting packages instead of a global setting. This should make it easier to maintain org-wise rules to allow scripts only for specific packages.
Is there a linter that could be used for scenarios like this to prevent unsafe default on package manager config?
Hi, this looks interesting, thanks for sharing. I am the builder of ingestr (https://github.com/bruin-data/ingestr), so I am very much in the same space.
I really like that you did this in Go, and I'll definitely dig a bit more into the source code to see how you tackled the CDC stuff, given that there is not many reliable CDC libraries in Go, and there are quite a few gotchas when it comes to doing CDC right. We also hand-rolled ours in ingestr, or I must say clanker-rolled, and we got quite a few things wrong in the first place.
Curious about the postgres-compatible query option: what's the usecase you have in mind there? My perception is that any org that would use Iceberg also has one or a few query engines in place, is this more for debugging stuff?
Zot seems interesting, this is the first time I see it. On a quickl look it seems like Pi, but in Go. I was hoping to embed Pi into some of our internal projects and the typescript stuff was blocking me, I'll definitely give Zot a look.
The project has been under development for over 6 months. We just open sourced it with a clean history. I am not sure what you expect here, should a project exist for months before it is worthy of a show post?
Thanks! DAC does that kind of validation partially, although doesn't validate the usage of the downstream dashboards. That's a very nice idea though.
In terms of validation it will validate queries, metric definitions, chart definitions and all ahead of time, before render. That way agents tend to validate their work much quicker.
I am not sure we are on the same page, as far as I am aware Vega doesn't do layout, does it? E.g DAC could use Vega for the charts and still take care of everything else around it.
There are quite a few libraries for charts and visualization, there are not as many for actually combining many of them with layouts, different components and including the actual implementation of the backend. Dac aims to provide all that as a standard and an implementation.
Hi, author here, thanks! I have used TypeScript before across various projects, but I haven't considered building CLI tooling in that before, I guess due to my prejudice against the whole JS ecosystem. I plan to give it another try in the next weeks.
I have been recently looking into extracting a bunch of details from a set of legacy invoice PDFs and had a subpar experience. Gemini was the best among the ones that I tried, but even that missed quite a bit. I'll definitely give this a look.
It seems like such a crowded space and there are many tools doing document extraction, I wonder if there's anything particular pulling more attention into the space?
This seems to undermine the engineering muscle these companies have. Fivetran is well-capable of building a query engine, and with this merger, they also get access to SDF's query engine. They have the engineering capabilities, as well as the capital to attract the talent where needed.
I would not underestimate any of these players in the space.
OpenFlow feels like an attempt at keeping customers within Snowflake boundaries, and while it might work for some, I see no way Snowflake being able to keep up with the data integration needs unless they allow another way of extending their capabilities other than pre-built integrations.
On the other hand, I do agree with you that it is quite a big challenge for Fivetran to try and become Snowflake.
I think consolidation in the space has been coming for quite some time now and this merger only confirms what us, along with many others, have been saying: the data tooling is in a miserable state and we had to glue together a bunch of different tools that don't work with each other.
At this point, I think it is quite obvious that Fivetran is going for Snowflake/Databricks's market share. They own the ingestion for many companies already, and they will offer a managed data lake product in order to compete with the data giants. By owning the means of bringing the data in (Fivetran) as well as the transformation layer (dbt/sqlmesh) they will aim to get ahead of Snowflakes of the world.
I think it'll be a win for the data community if they maintain and continue investing into the existing tooling, as they are running in quite a few places already, especially dbt core running in a self-managed way. I certainly hope they won't try to squeeze revenue for the sake of it from their combined users.
It's an interesting time to be in the space, and it feels great to be one of the few independent players in the market.