Correct, a common mistake people make is conflating these things. I wrote this several years ago about MongoDB:
One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated.
When talking about stores, it's important to be explicit about a few things:
1. Storage model
2. Distribution model
3. Access model
4. Transaction model
5. Maturity and competence of implementation
What happens is people talk about "SQL" as either an NSM or DSM storage model, over either a single node, or possibly more than that in some of the MPP systems, using SQL as an access model, with linearizable transactions, and a mature competent implementation.
NoSQL when most people refer to it can be any combination of those things, as long as the access model isn't SQL.
I work on database engines, and it's important to decouple these things and be explicit about them when discussing various tradeoffs.
You can do SQL the language over a distributed k/v store (not always a great idea) and other non-tabular / relational models and you can distribute relational engines (though scaling linearizable transactions is difficult and doesn't scale for certain use cases due to physics, but that's unrelated to the relational part of it).
Generally people talk about joins not scaling in some normalized form, but then what they do is just materialize the join into whatever they are using to store things in a denormalized model, which has its own drawbacks.
As to the comment above you, SQL vs NoSQL also doesn't have anything to do with the relative maturity of anything. Some of the newer non-relational engines have some operational issues, but that doesn't really have anything to do with their storage model or access method, it just has to due with the competence of the implementation. MongoDB is difficult operationally not because it's not a relational engine, but because it wasn't well designed.
Just like people put SQL over non-tabular stores, you can build non-tabular / relational engines over relational engines (sharding PostgreSQL etc.). In fact major cloud vendors do just that.
Cassandra is not a columnar database, columnar in this sense is about the storage layout. Values for a column are laid next to each other in storage, thus allowing for optimizations in compression and computation, at the expense of reconstructing entire rows. Postgres is a row store, meaning all the columns for a row are stored next to each other in storage, which makes sense if you need all of the values for a row (or the vast majority).
Having used both TSDB and ClickHouse in anger I have some thoughts on this:
They are both fantastic engines, I really like that both have made very specific tradeoffs and can be very clear in what they are good and bad at. Having worked on database engines, I can appreciate the complexity that they are solving.
My most recent use is with ClickHouse, which is great and I think a complete game-changer for the company. However there's a lot of issues (that are being worked on, the core team is great, though there are a few personalities that are a bit frosty to deal with). All of these comments come with love for the system.
1. Joins really need some work, both in the kinds of algorithms (pk aware, merge joins that don't do a full sort etc.), and in query optimizer work to make them better. We have analysts that use our system, and telling them to constantly write subqueries for simple joins is a total PITA. Not having PK aware joins is a massive blocker for higher utilization at our company, which really loves CH otherwise.
2. Some personalities will tell you that not having a query optimizer is a feature, and from an operational standpoint, it is nice to know that a query plan won't change, or try and force the optimizer to do the right thing. However, given #1, making joins performant (we have one huge table with trillions of rows, and a few smaller ones with billions) is really rough.
3. The operations story really needs some work, especially the distribution model. The model of local tables with a distributed table over it is difficult to work with personally. It would be nice to just be able to plug servers in without alot of work, like Scylla, and not have two tables that you have to keep schemas consistent with. THere's also just some odd behavior, like if you insert async into a distributed table, and only have a few shards, it'll only use a thread per shard to move that data over. It would be nice if there wasn't as much to think about.
4. Following #3, there's just too many knobs, maybe if they had a tuning tool or something that would help, but configuring thread pools is difficult to get right. I suspect CH could use a dedicated scheduler like Scylla's, that could dispatch the work, instead of relying on the OS.
5. The storage system relies a lot on the underlying FS and settings on when to fsync etc. I suspect if they had a more dedicated storage engine (controlled by the scheduler above), things could be more reliable. I still don't fully trust data being safe with CH.
6. Deduplication - This is a hard problem, but one that is really difficult to solve in CH. We solve it by having our inserters coordinate so that they always produce identical blocks, using replacing merge trees to catch stragglers (maybe), but it isn't perfect. A suggestion if possible is to try and put the same keys into the same parts, so they'll always get merged out by the replacing merge tree (I understand this is difficult).
The CH team is great, and these will be fixed in time, but these were the problems we ran into with CH.
TSDB was really solid, but we never used it at a scale where it would tip over. Our use case is really aligned with Yandex's so a lot of the functionality they have built is useful to us in a way that TSDB's isn't. (Also, being able to page data to S3 is amazing).
Generally these tend to be systems that work with machine generated data, my experience is with sensor data generated by automobiles (automated car efforts).
Naive solutions tend to either summarize the data, store as logs and then run batch processes to index in some form (or leave unindexed and just brute force the computation), or limit the incoming data rate to whatever could be indexed.
These can work for some use cases, but make it very difficult to operationalize these data sources (i.e use them to make real-timeish decisions).
Even human generated data sources (fb / twitter etc.) can generate something close to that data rate.
I've considered something like that, but instead of trying to figure out crimes, it would produce a score for bills.
A corruption score for bills, almost like a facebook for bills "This bill is friends with Exxon". It would figure out who spent the most getting the bill passed, and who they bought off to get it.
Just a simple thing for people to point to when they say things are corrupt. Granted in today's environment, that score would be 100% most of the time, but it would be interesting to have some idea just who bought the bill.
Honestly, if someone could mitigate the risk for this sort of thing, it's a huge deal.
Personally, I hate living in the tech hubs, I'd love to be able to move back to my hometown, it was a great place to raise kids, didn't have the same crushing traffic / monoculture and it was close to my family. I'm where I'm at due to the jobs, nothing else.
Most of them I'd wager. Issue is, and it was for me, is that moving out of an area where I can walk down the street and get another great job to an area where the only great job is the one that this hypothetical company is offering just wasn't worth the risk. I have a wife and kids, and so do most specialized senior engineers.
It depends on what you are doing. If you are trying to build something like a database kernel, you want people who have done it before, very specialized. There aren't lots of them outside of the tech hotspots.
I don't necessarily mean there aren't smart people outside of the hotspots, but there isn't the specialized skillsets that get built from working at the Microsofts / Googles etc. edit: In the number that you need.
Champaign obviously has lots of smart people (my parents went there), but you may have issues finding specialized senior people there.
We aren't going to agree on this, around remote employees. I believe there is value to co-located teams, not everyone agrees and that's fine.
The other point is, most startups do not need that talented of engineers, they think they do, but they don't. You don't want to pay bay area salaries if you don't have to, and in most of these areas, you don't have to.
If you are somewhere else but the bay, and you are paying people bay area salaries, but you aren't in SV, so you aren't raising VC like SV companies, I don't think that's a winning move.
If you are bootstrapping, and you are starting a company in an area that does not need the top tier of engineers (which is most of them, regardless of how they talk about hiring the best), I'd consider starting someplace low cost.
I think college towns in the midwest / rust belt are untapped resources, or areas that used to have strong technically focused companies that moved away. I've personally seen founders bootstrap and launch successful companies (B2B with real revenue, not Uber for cats) in areas with little tech presence.
The ability to not have to shell out huge salaries and equity was a real winner, there's also less distractions. You can get strong engineers (no, not SV / Seattle top end engineers, but people that can throw together a reasonable website) for < 100k in these areas, and they won't bounce around as much. You aren't competing with AmaFaceGoogSoft here, you're competing with HR companies and random consulting houses. There's disadvantages to being outside of the tech bubble, but advantages too. That being said, for an employee, you should at least try and do some time in SV / Seattle etc.
Caveat, this comment isn't directed at you (I agree with your comment), but rather the points around what you are saying.
One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated.
When talking about stores, it's important to be explicit about a few things:
1. Storage model
2. Distribution model
3. Access model
4. Transaction model
5. Maturity and competence of implementation
What happens is people talk about "SQL" as either an NSM or DSM storage model, over either a single node, or possibly more than that in some of the MPP systems, using SQL as an access model, with linearizable transactions, and a mature competent implementation.
NoSQL when most people refer to it can be any combination of those things, as long as the access model isn't SQL.
I work on database engines, and it's important to decouple these things and be explicit about them when discussing various tradeoffs.
You can do SQL the language over a distributed k/v store (not always a great idea) and other non-tabular / relational models and you can distribute relational engines (though scaling linearizable transactions is difficult and doesn't scale for certain use cases due to physics, but that's unrelated to the relational part of it).
Generally people talk about joins not scaling in some normalized form, but then what they do is just materialize the join into whatever they are using to store things in a denormalized model, which has its own drawbacks.
As to the comment above you, SQL vs NoSQL also doesn't have anything to do with the relative maturity of anything. Some of the newer non-relational engines have some operational issues, but that doesn't really have anything to do with their storage model or access method, it just has to due with the competence of the implementation. MongoDB is difficult operationally not because it's not a relational engine, but because it wasn't well designed.
Just like people put SQL over non-tabular stores, you can build non-tabular / relational engines over relational engines (sharding PostgreSQL etc.). In fact major cloud vendors do just that.
Correct, there's relatively easy ways to de-risk things. My wife and I dated for 3.5 years and lived together for 2 before we got married. We are in a high income bracket, and both of us thought quite a bit about the person we wanted to married, and made sure we had good conflict resolution skills, respect for each other. We agreed on kids, religion , money and sex (make sure you agree on these or you are going to have a bad time).
Chances of our marriage ending are very low according to the statistics. People always view marriage as this binary thing where you are either being completely and utterly scientific about things, or you are just being naive and jumping in. I love/d my wife deeply when we got married, but I was also aware of her flaws and my own. If you don't have any empathy or like living your life for yourself, don't get married (and probably don't date), it won't end well for you.
At the higher end of the market (i.e you can get hired at the majors in a senior role or equivalent at a smaller company), I generally see three types of people:
I see people that truly understand their market value, have data backing it up, they are professional (I always tell people how I appreciate the time / offer, and I'm polite to recruiters), demanding about information (I have little tolerance for ambiguity in compensation, define everything, this is how I pay the mortgage and save for retirement / my kids college).
Some managers think that is spoiled (not that you are necessarily saying that), but this is a business transaction, I ask for market and turn it down if it's not that, or if the compensation is ambiguous.
Then I see those that don't know their value and just sort of take whatever because they don't like interviewing / negotiating. I don't see many of those at the high end.
The third group are who you are talking about, total babies. They complain about recruiters, or how much everyone wants them, place crazy demands on companies (I worked with a guy who literally had a rider like Van Halen in his LinkedIn profile, he ended up being a total prima donna who got fired because everyone hated him). Most people in this third group think they are in the first, and they aren't worth recruiting, even if they are strong technically.
Sometimes people believe this means startups / small companies can't compete, but the issue is, startups / small companies really aren't offering any sort of market comp, they don't give enough equity to employees to make the risk worth your while. You don't necessarily need to pay me what AmaGoogFaceSoft do, but you need to give me a significant chunk of equity and give me enough visibility into how that is cut up (preferences etc.) in the company for me to value it, otherwise its worth 0, even if I believe that the company will be successful. If you can't trust me enough with that sort of transparency, then it's not gonna work out. With a huge public company, I know what my stock is worth roughly.
Amazon as a company believes margins are an opportunity, either for them, or for someone else. They don't believe in raising prices unless forced to by something external. You aren't correct here, they believe in making money on volume and breadth of services / products.
Anecdote incoming, but I've never seen that in 12 years in the industry (asking for an existing slip). And I've worked from everyone from AmaGoogFaceSoft types to HR companies to startups.
That's a bit of an oversimplification. Production grade RDBMS systems have far more guard rails, testing and work put in to them than Kafka. It's relatively straight forward to lose data in Kafka, I've done it (its usually control plane bugs, not data plane).
One thing that helps is if people stop referring to things as SQL / NoSQL as what ends up happening is various things get conflated.
When talking about stores, it's important to be explicit about a few things:
1. Storage model
2. Distribution model
3. Access model
4. Transaction model
5. Maturity and competence of implementation
What happens is people talk about "SQL" as either an NSM or DSM storage model, over either a single node, or possibly more than that in some of the MPP systems, using SQL as an access model, with linearizable transactions, and a mature competent implementation.
NoSQL when most people refer to it can be any combination of those things, as long as the access model isn't SQL.
I work on database engines, and it's important to decouple these things and be explicit about them when discussing various tradeoffs.
You can do SQL the language over a distributed k/v store (not always a great idea) and other non-tabular / relational models and you can distribute relational engines (though scaling linearizable transactions is difficult and doesn't scale for certain use cases due to physics, but that's unrelated to the relational part of it).
Generally people talk about joins not scaling in some normalized form, but then what they do is just materialize the join into whatever they are using to store things in a denormalized model, which has its own drawbacks.
As to the comment above you, SQL vs NoSQL also doesn't have anything to do with the relative maturity of anything. Some of the newer non-relational engines have some operational issues, but that doesn't really have anything to do with their storage model or access method, it just has to due with the competence of the implementation. MongoDB is difficult operationally not because it's not a relational engine, but because it wasn't well designed.
Just like people put SQL over non-tabular stores, you can build non-tabular / relational engines over relational engines (sharding PostgreSQL etc.). In fact major cloud vendors do just that.