The slides [1] helped me grok what problems this tool solves pretty quickly.
Everything big data is moving to blob storage these days, but streaming can lead to small files problem or longer latencies. File fragments stored locally with proxied readers seems like a simple solution to that.
Since duckdb is OLAP, it would directly compare with other columnar database technologies like Redshift, Presto/Athena, etc.
Most of these systems strongly encourage or outright enforce JSONL, so that’s the defacto standard, and most tooling or pipelines are going to generate that nowadays.
You can obviously still have a row of arrays, and different systems have slightly different approaches on how to deal with those. In Spark, this is referred to as “exploding”, in Presto you would cross join to unnest an array, in Redshift you can glob on the super type.
I’m not sure I have a particular favorite, only that the database support such an operation since it is a common occurrence.
Thanks for the recommendation! Now that you mention it, I’ve seen similar behavior on Dateline. Wonder if there are other terms for essentially “digging the hole deeper.”
Couple things that stood out to me was that the incident occurs in December and the raid ensues March 24th, so roughly 3 months. Building the case I presume.
Then after the raid, the accused doubles down and seeds fake news stories.
While certainly focused on larger vehicles, by all appearances Ford is much more exciting and innovative than its traditional domestic counterparts. The new electric F150, electric mustang, the Bronco series, etc. have all seemed to drum up quite the fanfare. I can’t say the same for GM.
Keeping in mind DE can mean different things at different companies, I spend a lot of time working on infrastructural components to just get at data reliably. Working in a product company with disparate generators of data, I’m often building out network connectivity (VPC peering, VPNs, etc.), subnets, ACLs, firewalls and load balancers across our visualization tools, managing job flows, controlling AWS costs, building read replicas for production databases, yadda yadda. There might be a ton of hoops to jump through before I can even start to process data and it’s the type of work that wouldn’t make sense to hand off to my DS counterparts.
If not a serverless model, then Docker at the very least. Something like Kubernetes might be overkill but containers make life easier both starting out and managing future growth.
Interesting, I’d guess list operations are significantly faster because of the cache, even a hash map like Redis? That can be a real slowdown in s3 with large quantities of files.
With direct integrations to EMR, Lambda, and Athena, its a great tool for building pipelines and effectively costs nothing on its own and is completely headache free.
Admittedly, its not always apparent. I think a general rule of thumb for me is if I can effectively write the script in my mind ahead of time. Anything beyond that and I might regret choosing a shell script for the purpose.
Sometimes a shell script just proves a concept out and is then rewritten later in a production language.
Shell scripts are obviously imperfect, but I don’t necessarily agree with this absolutism view. In the right circumstances, you can quickly prototype an idea or process information in a shell script with such low friction. Truly the hacker mentality, which I don’t think a “real” language can always provide in the spur of the moment.