How are you able to see a datasets lineage across storage types. For example how are you able to see that an s3 buckets files are the ancestor of some table in Postgres?
This idea reminds me of the classic video by google called the selfish ledger where they are able to create products based off of people’s direct interested and at the same time are able to influence society as a whole
Yes but I actually didn’t end up building any of the tile logic in python. I used PostGIS and Postgres. Each query is like 25 lines and supports polys and lines out the box.
Pulumi doesn’t have a framework for general CICD but from my experience it shifts the complexity out of the bash/yaml scripts and allows me to express it in python and the I can run unit tests and easily run it locally. Our use case is rather simple though, just a fast api backend and front on ECS.
I’ve been using Pulumi automation in our CI and it’s been really nice. There’s definitely a learning curve with the asynchronous Outputs but it’s really nice for building docker containers and separating pieces of my infra that may have different deployment needs.