I Can't Wait for NoSQL to Die (2010)(widgetsandshit.com)
widgetsandshit.com
I Can't Wait for NoSQL to Die (2010)
http://widgetsandshit.com/teddziuba/2010/03/i-cant-wait-for-nosql-to-die.html
46 comments
I felt this way too in 2010. Not so much now. 'Nosql' Data services like parse and firebase is how apps are going to be built. The speed from idea to implementation is a real game changer.
This is a good point of view. Honestly I never thought that services like Parse or Firebase could gain so much hype and get a place in the market for the lack of flexibility. But the truth is that only few apps are successful enough to require an high level of performance and flexibility, so it makes sense to build a first version using these services and eventually move away if the product is successful, there's no reason to build a top-class service to later have it barely or even not used at all...
I'd say the success of Parse and Firebase have a lot to do with the fact that they are services, especially public internet accessible services (in contrast to a host of DBaaS options that are meant for "behind the webserver" use). Maybe if someone created a version of MongoDB that could be safely accessed from the browser, it would have been just as interesting?
Or maybe it really is about the simpler APIs that Parse and Firebase provide.
Or, maybe Firebase is mega popular due to the realtime change events. Too many variables. :)
Or maybe it really is about the simpler APIs that Parse and Firebase provide.
Or, maybe Firebase is mega popular due to the realtime change events. Too many variables. :)
I would bet on development speed. Using SQL and No-SQL software brings a lot of engineering, you have to decide where to host, how to host, who is going to take care of that. This translates into costs. After that there's a new bunch of problems, it's required to engineers to know the platform and create the software for that. After that there's maintenance, etc... all things services like Parse and Firebase removed, with the trade-off of less flexibility, but as I said before, this can be a problem later in the development stage...
There are some areas where relational DBs simply have no place at all, and never had. For example, CADs always relied on the hierarchical database engines, and it should always stay this way.
This new hipster relational model is not nearly as universal as its proponents are claiming.
This new hipster relational model is not nearly as universal as its proponents are claiming.
Much like the relational model isn't universal, neither is the nosql model, and that's exactly the point. It keeps being adopted in places where sql is the right solution.
I'd consider it a kind of revenge. Previously we had to deal with an unwelcome invasion of SQL into the areas it's not suitable for, now the SQL guys have to fence off an invasion of document-, graph-, hierarchical- and whatever else models.
Once dust is settled on this battlefield, the world will become a better place, where more people are aware of their tools limits.
Once dust is settled on this battlefield, the world will become a better place, where more people are aware of their tools limits.
Especially when there is no such thing as "the NoSQL model". MongoDB, Cassandra, HBase, Redis etc have completely different ways of storing and retrieving data. And for those of us accessing these databases via Big Data technologies e.g. Spark/Hive they appear as 100% SQL databases.
The point here is that blanket statements are stupid. Especially when the line between NoSQL/SQL is completely grey now.
The point here is that blanket statements are stupid. Especially when the line between NoSQL/SQL is completely grey now.
I don't understand the part about NoSql replaced Ruby on Rails - apples were replaced by pears?
I like RDBMs, they get the job done for me (yay Postgresql), but his AdWords example is no longer correct. From Wikipedia:
> Eventually, Google developed a custom distributed Relational database management system (RDBMS) known as Google F1 specifically for the needs of the Ad business, which requires strong consistency, high scalability across data centers and powerful SQL queries. The interface has also been revamped to offer better work flow with additional new features, such as Spreadsheet Editing, Search Query Reports, and better conversion metrics.
> Eventually, Google developed a custom distributed Relational database management system (RDBMS) known as Google F1 specifically for the needs of the Ad business, which requires strong consistency, high scalability across data centers and powerful SQL queries. The interface has also been revamped to offer better work flow with additional new features, such as Spreadsheet Editing, Search Query Reports, and better conversion metrics.
It's no longer correct that it uses MySQL - it's still not using BigTable, and in fact an RDBMS seems to have been implemented instead.
That would appear to still support his rant?
You're right, that's a good point. And it seems Google has started to move away from BigTable with Spanner, which F1 is built on: https://en.wikipedia.org/wiki/Spanner_%28database%29
Lots of companies use NoSQL badly, and lots of NoSQL is implemented very badly. I can't help but think an awful lot of these engines read the bigtable whitepaper and completely misunderstand why it was important.
MongoDB, for example, uses collection latches. Thankfully, these are per-server, so if you're using sharding you've got one per server. It is still no better at single-server performance than MySQL using table-based locks (except that you are unable to write a query that uses two tables), except that the latches and 'documents' (locks and rows) are lighter-weight.
The writes in mongo are only document atomic. Why not use a latch per-document? It needn't be memory intensive; Use a bloom table to represent the latches, and a small amount of entries (a thousand or so) is plenty to allow real concurrency.
Lots of companies use collections badly. I've seen a collection of user addresses. These make no sense as a collection; They're less than a kilobyte long. They should be an array in the user object. Mongo documents are meant to be an order of magnitude larger than SQL rows, and a few kilobytes here and there won't break the bank. You're better off doing a 1M read than three serialized 1k reads; The RTT is comparatively long, even at sub-milisecond latencies.
MongoDB, for example, uses collection latches. Thankfully, these are per-server, so if you're using sharding you've got one per server. It is still no better at single-server performance than MySQL using table-based locks (except that you are unable to write a query that uses two tables), except that the latches and 'documents' (locks and rows) are lighter-weight.
The writes in mongo are only document atomic. Why not use a latch per-document? It needn't be memory intensive; Use a bloom table to represent the latches, and a small amount of entries (a thousand or so) is plenty to allow real concurrency.
Lots of companies use collections badly. I've seen a collection of user addresses. These make no sense as a collection; They're less than a kilobyte long. They should be an array in the user object. Mongo documents are meant to be an order of magnitude larger than SQL rows, and a few kilobytes here and there won't break the bank. You're better off doing a 1M read than three serialized 1k reads; The RTT is comparatively long, even at sub-milisecond latencies.
Not sure what you mean. MongoDB is not an implementation of BigTable whitepaper. It's a document store. And document level locking has been available in WiredTiger since version 3.0 and TokuMX since version 2.x.
And the most well known database that is derived from BigTable is Cassandra which is unquestionably very well designed and the authors (Facebook, Datastax) clearly understood what they were doing.
And the most well known database that is derived from BigTable is Cassandra which is unquestionably very well designed and the authors (Facebook, Datastax) clearly understood what they were doing.
Mongodb 3.0 with wiredTiger storage uses document level locking.
I felt the same when my previous company was stubbornly using MongoDB for its "flexibility" and we ran into all sorts of issues (following which we slowly migrated to Postgres -- having solid ORM libraries with similar interfaces helped).
I'm genuinely curious about one point, though: How is Walmart a "real business" vs Twitter?
I'm genuinely curious about one point, though: How is Walmart a "real business" vs Twitter?
> I'm genuinely curious about one point, though: How is Walmart a "real business" vs Twitter?
- One of them actually earns some money from time to time.
- If one of them disappeared overnight, it would have some impact on peoples' lives.There are scaling problems which require something that scales better than MySQL or Postgres. If you're Google, you really need BigTable. But, as I've pointed out before, unless your site has more traffic than Wikipedia, you probably don't have those problems.
Wikipedia is basically MySQL front-ended by ngnix and memcache caches.
Wikipedia is basically MySQL front-ended by ngnix and memcache caches.
Wikipedia is also unusually cacheable, a tiny fraction of users are even signed in. More dynamic and personalized content means hitting the limits of a traditional RDBMS with less traffic.
Stackoverflow runs on sql server. Even with dynamic sites, proper design tied to a proper rdbms scales quite nicely
MySQL and Postgres aren't sitting still on the scaling front, e.g.:
https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
True, but there are other RDBMS engines out there besides MySQL and Postgres.
It just seems that Oracle, Informix, SQL Server, DB2, Sybase SQL Server don't seat well with SV culture.
It just seems that Oracle, Informix, SQL Server, DB2, Sybase SQL Server don't seat well with SV culture.
They don't sit well with the SV culture in one simple (though extremely important) metric: They cost, and they cost a lot.
Everyone is assuming they will need to grow quickly. The costs of the systems you list become prohibitively expensive when you grow.
If e.g. Google or Facebook had to pay for an Oracle or SQL Server for their storage (in case those DBs could actually deliver the performance Google needs), or, for that matter - Windows Server licenses), they would have had to raise a lot more money and dump it into Oracle/MS. It is this that doesn't sit well with SV culture.
Everyone is assuming they will need to grow quickly. The costs of the systems you list become prohibitively expensive when you grow.
If e.g. Google or Facebook had to pay for an Oracle or SQL Server for their storage (in case those DBs could actually deliver the performance Google needs), or, for that matter - Windows Server licenses), they would have had to raise a lot more money and dump it into Oracle/MS. It is this that doesn't sit well with SV culture.
Even in places outside SV e.g. where I currently work, we're trying to move away from these solutions in some case. We have PB of data in DB2 and most core data goes through Oracle RAC along with SQL Server and Teradata.
But we still have use cases where we don't use these DB - partly for cost reasons and partly because we want different functionality. Berkeley DB ldap for accounts, ETL and analytics moving into Hadoop, some apps use Mongo and even other smaller apps use MySQL, and those are just the projects I know about from my view. We still pay for support contracts on all these projects but obviously costs are significantly lower.
Even in a company that doesn't mind paying large amounts for enterprise database, I don't think that that reaching for one of these major commerical DB is always the right solution. Not all projects need the support you're paying for from them, and beyond that for some useage patterns I don't think they're even the right solution to reach for, regardless the cost.
But we still have use cases where we don't use these DB - partly for cost reasons and partly because we want different functionality. Berkeley DB ldap for accounts, ETL and analytics moving into Hadoop, some apps use Mongo and even other smaller apps use MySQL, and those are just the projects I know about from my view. We still pay for support contracts on all these projects but obviously costs are significantly lower.
Even in a company that doesn't mind paying large amounts for enterprise database, I don't think that that reaching for one of these major commerical DB is always the right solution. Not all projects need the support you're paying for from them, and beyond that for some useage patterns I don't think they're even the right solution to reach for, regardless the cost.
Might be, on the markets I move on, their licensing is peanuts compared with overall project cost.
I always thought that Oracle was too expensive for pretty much everyone, but it says a lot when a Big Company dumps a Big Database vendor and moves to Oracle to save money.
It never saves money to move to Oracle, but it sometimes gives you functionality that PG/MySQL can't (e.g. high availability clusters).
> It says a lot when a Big Company dumps a Big Database vendor and moves to Oracle to save money.
I do not know which Big Company you are referring to? But also, never trust a headline which says "Big Company does X" to tell the whole story. It's very often part of a larger deal, a big discount given to them specifically for that headline, or some soft "nice business you have here, wouldn't it be awful if some patent litigation ruined it? That can be avoided if you buy our products" talk.
> It says a lot when a Big Company dumps a Big Database vendor and moves to Oracle to save money.
I do not know which Big Company you are referring to? But also, never trust a headline which says "Big Company does X" to tell the whole story. It's very often part of a larger deal, a big discount given to them specifically for that headline, or some soft "nice business you have here, wouldn't it be awful if some patent litigation ruined it? That can be avoided if you buy our products" talk.
I believe it has its places, just as SQL does. From what i have seen, NoSQL fits in where i was doing FS operations before, as a cache, or "blob" store. Somewhere to dump stuff for a while before its processed into SQL or whatever other format is needed. Something that can be easily replicated and queried instead of having to deal with the underlying FS and clustering that.
It also fits in as Key/Value stores for temporary session data, and where you need very high performance in memory access that can be shared between several servers or processes.
It also fits in as Key/Value stores for temporary session data, and where you need very high performance in memory access that can be shared between several servers or processes.
NoSql type of DBs have their value added. They'll not die, instead businesses will learn when they are useful, some the hard way some the easy way :)
still alive [2015] http://www.indeed.com/jobtrends?q=mongodb%2C+mysql%2C+couchd...
Is "times mentioned on a job advert" really an accurate measure of whether or not something is still used? Surely it lags years behind trends - if a technology is abandoned quickly there'll still be jobs maintaining the old code that need that skill. That doesn't mean the technology is still 'useful'. For example, there are still plenty of COBOL job ads around.
I'm not suggesting that's the case for NoSQL, just questioning the way you're measuring it.
I'm not suggesting that's the case for NoSQL, just questioning the way you're measuring it.
[deleted]
[deleted](1)
The sanctioned archive of TD's posts are here:
http://widgetsandshit.com/teddziuba/2010/03/i-cant-wait-for-...
http://widgetsandshit.com/dozba.html
http://widgetsandshit.com/teddziuba/2010/03/i-cant-wait-for-...
http://widgetsandshit.com/dozba.html
Thanks. We've changed the URL to that from http://www.memonic.com/user/dorian/folder/internet-tidbits/i....
No one is going to fund your start-up if you do not speak web scale. And the only web scale game in town is NoSQL. Also node.js.
I don't get if you are sarcastic or you really believe in what you wrote. I hope it's the first option.
Anyway, NoSQL is definitely a good solution for some usages, I personally created a special analytics service using Cassandra that is good to store unstructured and unrelated data to be later computed and analyzed. It would be barely impossible to achieve this without hitting on performances due to the data integrity checks that most SQL servers/engines perform. On the other hand, I use SQL everywhere else and I am happy with that. It scales well if the design is well made and it's just a matter of research and a little bit of preparation to create an SQL environment that scales. It's not easy as some NoSQL databases, but not that hard and impossible as some of these solutions are claiming to gain new customers...
On Node.js I have some doubts, even if the general nature is very scalable and is well proved as fact, there are other solutions that are good alternatives like Golang or some frameworks written in Scala, maybe on top of AKKA.
The big problem is that a lot of companies and investors are asking for scalability, but the reality is just that one service on hundreds built are going to be so successful to require a scalability plan. If you hit that target you would definitely get enough money to even migrate the technology to something better and scalable.
Anyway, NoSQL is definitely a good solution for some usages, I personally created a special analytics service using Cassandra that is good to store unstructured and unrelated data to be later computed and analyzed. It would be barely impossible to achieve this without hitting on performances due to the data integrity checks that most SQL servers/engines perform. On the other hand, I use SQL everywhere else and I am happy with that. It scales well if the design is well made and it's just a matter of research and a little bit of preparation to create an SQL environment that scales. It's not easy as some NoSQL databases, but not that hard and impossible as some of these solutions are claiming to gain new customers...
On Node.js I have some doubts, even if the general nature is very scalable and is well proved as fact, there are other solutions that are good alternatives like Golang or some frameworks written in Scala, maybe on top of AKKA.
The big problem is that a lot of companies and investors are asking for scalability, but the reality is just that one service on hundreds built are going to be so successful to require a scalability plan. If you hit that target you would definitely get enough money to even migrate the technology to something better and scalable.
This is where a bunch of people with no clue talk about how great pgsql is and everyone should create their startup using raw sql and only psgsql because it does everything even nosql ;) It is funny how forums gravitate towards inappropriate and obscure solutions and then recommend them in every situation. You can see the same effect all over the internet. Eg MFA on reddit recommending Allen Edmond shoes. A completely inappropriate shoe with leather soles that costs 10 times as much as modern shoes.
NoSQL databases are much nicer to work with for typical web applications (yeah, even the ones with relational data). They are much closer to how we actually use data as a programmer. Most startups using a SQL database will be using an ORM, which is a massive piece of software designed to make SQL databases more compatible with how we actually work. Even then, they still don't have the flexibility you get from using a nosql solution.
As for this article, it is hard to get past the fact that he thinks the only reason to use nosql is for scalability. He also states that mysql was fine, yet fails to understand that the massive startups using mysql didn't use it at all like a relational database.
NoSQL databases are much nicer to work with for typical web applications (yeah, even the ones with relational data). They are much closer to how we actually use data as a programmer. Most startups using a SQL database will be using an ORM, which is a massive piece of software designed to make SQL databases more compatible with how we actually work. Even then, they still don't have the flexibility you get from using a nosql solution.
As for this article, it is hard to get past the fact that he thinks the only reason to use nosql is for scalability. He also states that mysql was fine, yet fails to understand that the massive startups using mysql didn't use it at all like a relational database.
> They are much closer to how we actually use data as a programmer.
No, they're how you, as what sounds like a predominantly object-oriented programmer, use data. And that's fine, but don't pretend it's universal. A programmer approaching a problem from a functional perspective may well exploit the relational nature of an RDBMS to work smartly within the paradigm. (I've seen it done, and done very well and easily, in a way that's approachable even for relatively novice functional programmers.) Even if using an object-oriented style of programming, with the feature set of something like PostgreSQL--or SQL Server, if that's your bag--and the ability to very easily provide assurances around data integrity and correctness using the built-in tools, you have a very powerful and easy-to-use set of tools for building a rugged, correct data model. For product developers that do not find a cavalier approach to users' data to be acceptable, this can be a wise choice. Being fast and wrong is not always the right answer (and I personally consider it to usually be the wrong one).
But it's even better to be fast and right, and that can quite obviously be done, in many cases, with a relational data store. And this may mean understanding how to write code in a style that doesn't necessarily use the Active Record pattern or a similar ORM. That's true. But that does not imply an inherent product development benefit to using such a tool (for their partisans), nor does the common reliance on these tools and its well-documented failures imply that the various NoSQL technologies are better at building a product.
No, they're how you, as what sounds like a predominantly object-oriented programmer, use data. And that's fine, but don't pretend it's universal. A programmer approaching a problem from a functional perspective may well exploit the relational nature of an RDBMS to work smartly within the paradigm. (I've seen it done, and done very well and easily, in a way that's approachable even for relatively novice functional programmers.) Even if using an object-oriented style of programming, with the feature set of something like PostgreSQL--or SQL Server, if that's your bag--and the ability to very easily provide assurances around data integrity and correctness using the built-in tools, you have a very powerful and easy-to-use set of tools for building a rugged, correct data model. For product developers that do not find a cavalier approach to users' data to be acceptable, this can be a wise choice. Being fast and wrong is not always the right answer (and I personally consider it to usually be the wrong one).
But it's even better to be fast and right, and that can quite obviously be done, in many cases, with a relational data store. And this may mean understanding how to write code in a style that doesn't necessarily use the Active Record pattern or a similar ORM. That's true. But that does not imply an inherent product development benefit to using such a tool (for their partisans), nor does the common reliance on these tools and its well-documented failures imply that the various NoSQL technologies are better at building a product.
Err no. It's great that you have just discovered functional programming but your claim that the data presented by a relational database is only unnatural for object oriented programming is absurd. When you join tables you get something like this:
User1 firstname1 lastname1 role1 rolename1
User1 firstname1 lastname1 role2 rolename2
User1 firstname1 lastname1 role3 rolename3
This is purely a result of how the database works. That data is never presented to the user in that way. It has to be converted into a new form.
User1 firstname1 lastname1 role1 rolename1
User1 firstname1 lastname1 role2 rolename2
User1 firstname1 lastname1 role3 rolename3
This is purely a result of how the database works. That data is never presented to the user in that way. It has to be converted into a new form.