"Here’s why: There is absolutely no benefit for you to gain by talking in an exit interview, and plenty of negative consequences to come out of it. At best you’ll be remembered as a complainer, and you may make enemies."
I guess I would counter with if I have friends there, I would like their lives to be better. If my exit interview is able to do that, then I would take that as a net positive.
> RDBMS based data warehousing is obviously not economical at the petabyte scale.
I never said it was .. I'm simply responding to "I simply cannot understand how anyone chooses this over running your own Spark clusters with Jupyterlab". I'm trying to help you understand why folks would choose a SaaS over run your own.
> But storing all that data in S3 with Delta Lake/Iceberg format and querying with Spark changes things entirely. You only pay for object storage, and S3 read costs are trivial.
No. You don't just pay for object storage + minor S3 read costs.
You pay for operations
You pay for someone setting up conventions
You pay to not have to optimize data layouts for streaming writes
You pay to not have to discover race conditions in s3 when running multiple spark clusters writing to same delta tables
You pay to not have to discover that your partitions/clustering needs have changed based on new data or query patterns
But look .. I get it. You have chosen to optimize for cost structures in one way .. and I've chosen to optimize in a different way. In the past I've done exactly as you've said as well. I think being able to seeking to see _why_ folks may have chosen a different path may help you understand other areas to consider in operations.
Here was my situation. Occasional queries. Over a couple petabyte of data. Customer facing so response in seconds per SLA but >
95 percent of the time the warehouse isn’t running. Cached queries from within 24 hours which don’t require the warehouse to even spin up. Our snowflake costs were significantly less than an FTE.
Would that potentially be a situation which “running your own” doesn’t make sense?
“I can’t imagine being a traditional teacher, and having to teach kids that don’t care.”
What I’ve seen is igniting that spark and moving folks from not caring to caring (or realizing they can affect change) can be even more rewarding than the other mentoring being discussed
Let’s say you work for a SaaS doing analytics. Your boss says “hey! We need to start reporting on new logos. Can you snag those from the DB?”
But what counts as a new logo? Does a pro serve engagement that doesn’t use the product count? What about a business using the SaaS but still in a trial period? Etc.
A semantic layer helps provide common agrees upon definitions to the business. So any one looking for common data entities can just look those things up… and can come to published definitions (which are backed by queries to databases, data lakes, etc).
Does that help? Another example of this would be dbt for example
Yeah. The frameworks solve the 80 percent problem quite well but once you start trying to be as specific as the business often wants, I would find them lacking and end up popping JSON to d3 (or some such)
There's a lot of ... sub par BI devs who want to point and click through horrendous PowerBI interfaces and such. Personally i find that style to hamper my ability to express what the business units need out of data products.
It's just considered "data" these days. We just look at the Vs of the data and adjust based on those. High velocity? Do X. High volume? Accommodate Y. High variety? ... The other side of things is the underlying data quality often had tons of issues, so there's been a lot of focus on the data observability (which isn't sexy at all).
Still tons of folks out there using Hadoop (ew), Snowflake, etc. New technologies coming out include things like Trino, Apache Iceberg, etc. So it's there ... just no one cares about the moniker .. just getting things done.
If I recall, the language that's used in situations like these is something along the lines of "Personal electronic devices will be governed under this policy when such devices are attached to the CPS network." So .. it's a "as long as you're going to connect to the CPS controlled network / resources, you must have software protections / controls in place".
It’s probably worth going somewhere specifically for testing giftedness so you can see if your child (on that day) is more likely gifted, highly gifted, profoundly gifted, etc. Also it can be helpful to find out if there’s other 2E (twice exceptional) things at play such as ADHD, etc.
We took our kiddo on a trip to Colorado to the Gifted Development Center which was a good helpful first step (and connected a lot of dots for us). More than happy to share experiences at matt at mindvault dot com.
Modified air packaging with carbon monoxide is _very_ common in the US. So the “artificial color” isn’t some dye or something… it’s how the myoglobin reacts to the MAP.
I guess I would counter with if I have friends there, I would like their lives to be better. If my exit interview is able to do that, then I would take that as a net positive.