Totally agree with Redshift sentiments. It's been lovely seeing BigQuery and Redshift step their game up over the past 1.5yrs, because they really should have been doing certain things for many years prior.
Re: Firebolt, I don't consider it to be in the same class as Snowflake whatsoever (even though their advertising seems to indicate otherwise). Snowflake is like a very powerful swiss army knife. Firebolt is good for a very specific (dare I say niche?) workload but falls all over itself for the vast majority of data org needs.
Ah yes, the common misconception that trades are simplistic!
Most trades (!including plumbing!) have VASTLY more stringent requirements than tech, are often more mentally stimulating, and yes, quite rewarding.
Disclaimer: I've worked in software for ~9 years and worked in trades before that. ASE master certified auto technician, father is a master electrician, uncles are contractors. I'll probably become a plumber.
PipelineDB was awesome - built some v. neat things with it 4-5 years ago. Wouldn't use it now since the team went to Confluent. https://ksqldb.io/ functionality is getting close.
Flink is a solid option. Materialize shows a ton of promise.
Nice post! It's always fun reading about people being creative and challenging the analytics status quo (aka GA). Besides the joy of doing it yourself, you've accomplished a couple other things worth mentioning:
1. You'll never be sampled. GA samples historical data pretty heavily, and you have to pay for 360 to retain unsampled event data (at a tune of $160k+ per year).
2. You have full access to all generated data.
I'd highly recommend using Snowplow's javascript tracker (https://github.com/snowplow/snowplow-javascript-tracker) in a very similar manner to what you've outlined here. You'll get a ton of extra functionality out of the box, which would add yet another level of insight. With snowplow, you get the following for free:
1. Sessionization, which is consistent with google analytics' definition - effectively a 30 minute window of activity.
2. User identification - the tracker drops a persistent cookie (just like GA), so you can see returning visitors.
7. Ability to make your event tracking 100% first-party
(Disclaimer: I don't work for them, but I've seen the system work very well a number of times.)
I'm running a similar setup on my blog, and it costs well under $1 per month: https://bostata.com/client-side-instrumentation-for-under-on.... I'm doing the same exact thing with Cloudfront log forwarding and have several lambdas that process the files in S3. From there, I visualize traffic stats with AWS Athena (but retain a ton of flexibility, since they are all structured log files).
Props for diving into such a hot topic though. This is a really big deal for companies of many sizes (probably _all_ sizes), and bringing creativity to this conversation is +1
Inefficient sql? Crank the virtual warehouse.