Python setuptools 36.0; still broken, 9 hours later
github.com2 pointsby shadowmint3 comments
- All the data in one place.
- An easy way to explore the data.
- A single source of truth for transformed data.
- Metadata to explain the data model (ie. documentation).
- Adds yet another maintain-forever technology to your stack.
- Adds yet another pipeline (or set of pipelines) that does the same thing.
- Moves from an architecture that is clustered for scale (ie. spark) to one that only scales vertically (postgres).
- Potentially introduces *yet more* sources of truth for some data. - it doesn't scale
- SQL is terrible language to write transformations in (its a *query* language, not an ETL pipeline)
- it's only vaguely better when you have many denormalised tables, vs. s3 parquet blobs
- you have to invent data for schema changes (ie. new table schema, old data in the table) (ie. migrations are hard)
That's not what they wanted.
What people are being sold is AI/ML as a magic bullet that will do something useful regardless of the situation, and it lets business people avoid making decisions about what they actually want, because AI/ML can be anything, so they just signup for it and expect to get 20 things they didn't know they wanted handed to them on a plate.
Turn out, it's not enough to just collect a bunch of data and wave your magic wand at it. It wasn't with web analytics 10 years ago, it's still not.
What you actually need is someone who has a bunch of tricks up their sleeve, and has done this before, and can suggest a bunch of Business Insights the business might need before they start building anything, people that actually decide what to do, and actions taken to investigate, and solve those problems.
I mean, to some degree you're right; perhaps ML models could be useful for tracking hardware failures, but that's not what the parent post is talking about. The previous post was talking about just collecting the data and expecting the predictive failure models to just jump out magically.
That doesn't happen; it needs a person to have the insight that the data could be used for such a thing, and that needs to happen before you go and randomly collect all the wrong frigging metrics.
...but hiring experts is expensive, and making decisions is hard. So ML/AI is sold like snake-oil to managers who want to avoid both of those things. :)