Rather than answering directly, I'm thinking about this problem from the other end altogether ever since I saw the dbricks rt demo. Apologies for the rambling response, as I haven't yet finished thinking about this problem...
We ended up with 'hot' data in oltp and 'cold/archival' data in olap because the storage size of oltp has always been limited.
(1) Limited by computation - there's only so much data that we can store on disks and nvme
(2) Limited by wallet - disks and nvme are EXPENSIVE
Also, the tight coupling of compute and data didn't help. It limited the size of databases on the individual expensive compute nodes.
So, another question will be -
What's currently stopping me from keeping the scd history tables right in my oltp db? what's forcing me to copy state into my etl/elt pipeline and the process it into scd into a dedicated olap db?
To some extent,the answer is still the same - the oltp cannot scale for the storage size required for keeping historical data. So, I've had to take out the 'cold' historical data and keep it in my olap freezer.
Now, if oltp itself is scaling, I'm not gonna bother with the copying step. I'll just prefer to store the history in oltp itself.
In my perspective (majorly from handling IoT systems), I need olap for 2 reasons - (1) storage scalability, and (2) analytical processing speed
I now consider (1) to be a solved problem
As for (2), I'm still not sure how this architecture ends up matching the query processing speeds of column-oriented storages. But again, I need to study more.
The SCD pipeline still remains in some form. Either in the form of (1) scd rows that we currently keep (etl pipeline)
, or (2) as older lsn rows that simply don't get deleted (existing db engine).
I've done quite a lot of experimentation with (2), and it is a pretty solid concept to work with.
I've spent quite a lot of years hammering my brain at databases and datastores in general. And I've now got a feeling that this is it.
Finally.
Data engineering and systems design are incredibly close. After some time, they completely mesh together.
And from personal experience, the knowledge-volume to become practically potent is actually smaller from data engineering side (than from normal software development side).
I've found it easier to get data gigs and simultaneously strengthen the knowledge.
Beyond the theory, if you are having trouble getting your hands dirty, here is the path I had taken for data engineering:
(It is pretty hands on.)
After you are done with the initial learning, find an academic machine learning discord or something similar. There will always be people there who will be very happy to find someone to clean their data. It's a great way of getting hands on with data engineering.
System design is best learnt through fires.
A good angle of attack is: pick a certification like AWS SAA or equivalent(AWS, azure, Google cloud. Doesn't really matter. Just pick a mid level certification).
Then do the labs. They will quickly point out the holes in your knowledge/understanding. The free tier will take care of your needs, and cloud providers most of the time forgive the first surprise bill(it happens to everyone).
Soon data engineering and systems design merge anyways. This path is like a cheat-code for forcing convergence. Otherwise, the mind will quickly forget most theory.
---
Adding to this:
When you get hands on, you will find how the mind lies to you about how much you know.
Mind maps are a nice way to reliably detect holes in understanding. Sit down with paper & pen once a week or so, and make a big map of everything you know.
Web development is hard. The individual parts are manageable, but bringing them together is a mess. We web developers are aware of this. It's like an open secret.
And it takes time getting comfortable with the mess, and letting the brain develop its own map of how things work. And you will do stupid mistakes along the way. Those are funny actually.
I'll give you my current perspective:
I've been at this for a decade, and I am definitely not incompetent. Last week I spent the night fixing a seemingly simple bug. It ate up my night and sanity. I went to bed angry. Woke up, are realized that I was looking at the right service, but editing the files in the wrong folder. Of course I didn't see the output. And I couldn't stop laughing.
This sort of silly mistakes happen almost once every month.
It can be frustrating, it will be. In webdev, things are in constant flux.
But try looking at it through a sense of humor, and you will see how funny it all is. The mentality will carry you through the decade.
We are just anxious & weak monkeys, with big brains, worrying about glowing glass powered by stone with lightning. It's surprising we even made it this far.
You and I deserve the leeway to make some mistakes for ourselves.
All right, this one is not purely technical. It's technical, but mixed with comics, art and a lot of personality.
It is an old classic in the community, and something that I aim up towards. Opened up my imagination to what a unique thing a technical book can become.
Mindfulness is something that emerges as a side-effect from meditation. Not something that you actively aim for. I'm going to have a lot of trouble putting it into words, since this is something I got from culture, and not from specific words. Bear with me.
Here is how it feels: the brain experiences stuff "objectively".
For example: if something happens and you get angry, instead of fully being angry, you get to see that you are angry. And now you get the choice to be angry or not. Or sometimes you don't get a choice, but can clearly see it and realize that it's something you have to go through. There emerges a clear separation between your brain, your emotions and the core you.
It's not something you do consciously, it's something that your brain does by itself all the time. Like fish noticing water. Things don't really change, but you get a perception to notice and see things.
An analogy I like to use is: You are playing a 2d game. Your character exists in a 2d world. But your perspective is 3d and can clearly see what's happening.
--------
P.S.
Since you seem to be curious, keep in mind that meditation is serious stuff and needs to be given respect. Using meditation for just feeling good is like using a gun to dig a hole. Can you do it? yeah. But without gun discipline, you may end up shooting yourself in the foot.
Meditation is a tool. It exists safely only within a larger ecosystem of other things. Prescribing meditation openly without the ecosystem is dangerous.
Once mindfulness comes, the problems get more and more subtle.
By the very nature of solutions: a new company cannot beat google at general search. Google was and is search.
Like a tree that has grown up and shadows the entire land around it. There will be random small pockets of sunlight with smaller plants in it. But the big tree owns the area.
There can be a short term competition via extremely specialized search engines, but they will not rise to the same dominance. Google, in a way, is the entire idea of searching the web.
The next dominant generation has to be from a new paradigm, that makes web search obslete.
---
A relatable current example could be: youtube being dwarfed by tiktok. The nature of the new thing is such that it very naturally dethrones the old thing, without directly competing with the incumbent.
There will be specialized providers like vimeo, But at this point, youtube IS the idea of video on the internet.
---
Just note the context of this understanding: This understanding came from struggling to change the education system of my country. And then it seemed to apply everywhere I saw.
I'll round-off my reply with this quote:
“You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.”
I ended up using Pulsar music player, only because we were camping and a friend send me the apk via bluetooth (I was a spotify person before I started heavily going out).
It simply worked for my old folderwise-kept mp3s etc, and it also displays embedded lyrics in the mp3 files which was a killer feature.
It's not that I love this app dearly, it's just that it got the job done and didn't need replacement.
We ended up with 'hot' data in oltp and 'cold/archival' data in olap because the storage size of oltp has always been limited.
(1) Limited by computation - there's only so much data that we can store on disks and nvme
(2) Limited by wallet - disks and nvme are EXPENSIVE
Also, the tight coupling of compute and data didn't help. It limited the size of databases on the individual expensive compute nodes.
So, another question will be -
What's currently stopping me from keeping the scd history tables right in my oltp db? what's forcing me to copy state into my etl/elt pipeline and the process it into scd into a dedicated olap db?
To some extent,the answer is still the same - the oltp cannot scale for the storage size required for keeping historical data. So, I've had to take out the 'cold' historical data and keep it in my olap freezer.
Now, if oltp itself is scaling, I'm not gonna bother with the copying step. I'll just prefer to store the history in oltp itself.
In my perspective (majorly from handling IoT systems), I need olap for 2 reasons - (1) storage scalability, and (2) analytical processing speed
I now consider (1) to be a solved problem
As for (2), I'm still not sure how this architecture ends up matching the query processing speeds of column-oriented storages. But again, I need to study more.
The SCD pipeline still remains in some form. Either in the form of (1) scd rows that we currently keep (etl pipeline) , or (2) as older lsn rows that simply don't get deleted (existing db engine).
I've done quite a lot of experimentation with (2), and it is a pretty solid concept to work with.
I've spent quite a lot of years hammering my brain at databases and datastores in general. And I've now got a feeling that this is it. Finally.