The durability and transformation reasons are definitely more compelling, but the article doesn’t mention those reasons.
It’s mainly focused on the insert batching which is why I was drawing attention to async_insert.
I think it’s worth highlighting the incremental transformation that CH can do via the materialised views too. That can often replace the need for a full blown streaming transformation pipelines too.
IMO, I think you can get a surprising distance with “just” a ClickHouse instance these days. I’d definitely be interested in articles that talk about where that threshold is no longer met!
Sure, but the article doesn’t talk about that, it seemed to be focused on CH alone, in which case async insert is much fewer technical tokens.
If you need to ensure that you have super durable writes, you can consider, but I really think it’s not something you need to reach for at first glance
Not OP, but to me, this reads fairly similar to how ClickHouse can be set up, with Bloom filters, MinMax indexes, etc.
A way to “handle” partial substrings is to break up your input data into tokens (like substrings split in spaces or dashes) and then you can break up your search string up in the same way.
Claiming delay compensation if you don’t have intent to travel is the fraud part.
Easiest example is if you have a season ticket, but you have the day off. You weren’t going to take the train to work that day, so no intent to travel. If you claim DR, then that’s fraud for the compensation.
A small question on the schema, I noticed that you have only “_now” as the Order By (so should just use that for the primary key). Do you expect any cross tenant queries?
Just my feeling would be that I’d add the tenant ID before the timestamp as it should filter the parts more effectively
TBH, as the Dockerfile based Lambda layers have a 20GB file size limit, I'm not sure the size matters as much.
It's certainly easier to figure out what's going on in a smaller container though! I've had to debug some nasty situation with layers in Python Lambdas before and it's not fun...
I used BPF (using _both_ of Brendan Gregg's recent books!) on our Jenkins builds recently to figure out why `TRUNCATE` statements were taking longer than I expected.
The underlying cause was that `ext4` filesystem was journalled and the `fsync`s were waiting on `jbd2_log_wait_commit`, which offcpu sampling let me pick up.
I don't think I would have been able to trace the kernel call and link it all the way back to the application call without BPF (at least I wouldn't have been able to).