Predictions on the future of databases(gigaom.com)
gigaom.com
Predictions on the future of databases
http://gigaom.com/2013/12/14/5-predictions-on-the-future-of-databases-from-a-guy-who-knows-databases/
10 comments
Interesting - thanks for sharing that. It highlights the disconnect between the theoretically correct solution and the practical one.
The object-relational mismatch is still a huge problem. I'm quite interested by this supposed solution. But I spent 5 minutes chasing links and couldn't find an example of what the syntax actually looks like, only a suggestion that I buy some book to understand the language. This "D" thing sounds pretty cool, but it seriously needs some more user-friendly marketing (and a non-colliding, googleable name).
Tutorial D is an example language spec, but the syntax itself isn't really the point.
D would overcome the object-relational mismatch by making the relational side much, much more powerful. It's really about unifying the ideas of "tables of data are relations" and "relations are logical propositions", so that one could have an über-powerful general purpose language that just happens to also be a natural interface for databases.
D is essentially a set of low level prescriptions that the authors felt would be required in order to achieve such a language.
D would overcome the object-relational mismatch by making the relational side much, much more powerful. It's really about unifying the ideas of "tables of data are relations" and "relations are logical propositions", so that one could have an über-powerful general purpose language that just happens to also be a natural interface for databases.
D is essentially a set of low level prescriptions that the authors felt would be required in order to achieve such a language.
Most of theninterestin info is linked from the reference material section of www.thethirdmanifesto.com; what you Semmelweis to be looking for is:
http://www.dcs.warwick.ac.uk/~hugh/TTM/Tutorial%20D-2013-05-...
Which provides a description of Tutorial D, an implementation of the prescriptions.
http://www.dcs.warwick.ac.uk/~hugh/TTM/Tutorial%20D-2013-05-...
Which provides a description of Tutorial D, an implementation of the prescriptions.
> ... Semmelweis ...
Or "seem" -- I have really no idea how that happened and didn't notice it until too late to edit.
Or "seem" -- I have really no idea how that happened and didn't notice it until too late to edit.
That's further than I managed to get, but still seems to be a language spec rather than a tutorial.
There's a lot going on and that's the way it has been for quite a while now but as someone who's working with databases only marginally at the moment, I'm left wondering: What's wrong with RDBMS? Sure it's nice to have a simple Key-Value Store for applications that don't need much more than this but bottom line, the relational model is very powerful, albeit quite abstract, and most data we need to deal with can be made to fit into this model. We have a working, declarative query language that Just Works(TM), for which we have written very good optimizers. The various instances of such DMBS range from small-scale use (for example, SQLite) to really-huge (Postgres, Oracle, etc). So, to sum it up: Exactly why should we abandon this?
> the relational model is very powerful, albeit quite abstract, and most data we need to deal with can be made to fit into this model
Maybe, but it's awkward. Graph or tree structures are still painful to store in relational databases, and a lot of problems turn out to involve those shapes.
> We have a working, declarative query language that Just Works(TM), for which we have written very good optimizers.
Maybe, but the tooling is still terrible. Where are the libraries? Where are the IDEs? Where's the integration that makes it easy to call procedures in my application language from SQL and vice versa? General-purpose programming languages have got better and better in the last 25 years, while SQL has stayed static.
> So, to sum it up: Exactly why should we abandon this?
Look at the ORM problem. Why do people continue to use these horrific bloated, leaky tools? Because it's really nice to be able to store and query in the native language of your application. We need storage tech that's better at supporting this.
Maybe, but it's awkward. Graph or tree structures are still painful to store in relational databases, and a lot of problems turn out to involve those shapes.
> We have a working, declarative query language that Just Works(TM), for which we have written very good optimizers.
Maybe, but the tooling is still terrible. Where are the libraries? Where are the IDEs? Where's the integration that makes it easy to call procedures in my application language from SQL and vice versa? General-purpose programming languages have got better and better in the last 25 years, while SQL has stayed static.
> So, to sum it up: Exactly why should we abandon this?
Look at the ORM problem. Why do people continue to use these horrific bloated, leaky tools? Because it's really nice to be able to store and query in the native language of your application. We need storage tech that's better at supporting this.
Most boots on the ground DBAs don't know anything about rational DB design. Codd Normal form? Whats that? Can't I just make one giant table? Indexes? Why would I want indexes? I heard they take up disk space and CPU so I won't use any, to increase performance. So they (poorly) implement key value stores regardless what the problem actually requires (if the only tool you know how to use is a nail...). Nothing funnier that watching a noob do a select * on a large table transferring gigs of data and use his RoR app to implement the WHERE using nested ruby if statements and string comparisons (or better regexes without understanding regexes, LOL) and god help me a bubble sort instead of an ORDER BY clause and of course hit another table once for each row selected from the first table because he doesn't know JOINs even exist, or heard they're "slow".
A RDBMS is a very slow key value store, so anything that can't do transactions or can't do anything else, is always going to be faster at its extremely limited set of abilities.
From an engineering perspective if you need 10 HP to run your 5 KW generator, a RDBMS is like installing a 10000 HP marine diesel and then not having any idea how to start or maintain it or even where to get the fuel. Obviously every objective performance metric would be better for a 10 HP lawnmower engine in that app, if all you'll ever plan to use is 10 HP and you have no idea how to use anything more advanced anyway.
There are also very loudly trumpeted anecdotal situations or contrived thought experiments where certain unusual technologies fit a unusual situation very well. This is, oddly enough, unusual.
I've read and experimented with the "7 DBs in 7 weeks" book and it IS very interesting but I can't find any business cases to actually use any of it, which is somewhat frustrating. And my experience is how you end up with people writing CRUD apps to store cooking recipes that none the less use NEO4J because they really, really, want to add a line to their resume that they used NEO4J, not because the app needed it.
A RDBMS is a very slow key value store, so anything that can't do transactions or can't do anything else, is always going to be faster at its extremely limited set of abilities.
From an engineering perspective if you need 10 HP to run your 5 KW generator, a RDBMS is like installing a 10000 HP marine diesel and then not having any idea how to start or maintain it or even where to get the fuel. Obviously every objective performance metric would be better for a 10 HP lawnmower engine in that app, if all you'll ever plan to use is 10 HP and you have no idea how to use anything more advanced anyway.
There are also very loudly trumpeted anecdotal situations or contrived thought experiments where certain unusual technologies fit a unusual situation very well. This is, oddly enough, unusual.
I've read and experimented with the "7 DBs in 7 weeks" book and it IS very interesting but I can't find any business cases to actually use any of it, which is somewhat frustrating. And my experience is how you end up with people writing CRUD apps to store cooking recipes that none the less use NEO4J because they really, really, want to add a line to their resume that they used NEO4J, not because the app needed it.
Condescend much?
When I'm right, yeah. Not using "engineer" as a title of authority but as a problem solving technique, maybe the TLDR is its an engineering problem where non-engineers mess up RDBMS incredibly often, and when its blame time, better off blaming the tool than the "designer" so...
"I don't know what I'm doing, but someone who knows what they're doing anecdotally solved a completely different problem using tool XYZ, so for lack of any better idea, lets copy them".
If you're familiar with cargo cult science there is an enormous miasma of cargo cult engineering fogging up the entire database arena not just nosql.
Another concept that needs to be in the discussion is the "no silver bullet" rule from programming applies to database design, like it or not. Can't just sprinkle magic nosql pixie dust on any old random problem and expect it to work, any more than applying any random programing fad to any random problem will work.
The (old and new) tools are actually pretty interesting, although often poorly engineered (by the end user) and implemented. Its the persistent anti-patterns and non-engineering design technique that I'm properly arrogant and condescending toward.
Hammers are a cool new invention and have some great unusual new applications, but they don't install deck screws any better than the legacy screwdriver. Laborers on the job randomly mixing screws nails screwdrivers and hammers on the job and then internet discussions about how hammers and/or screwdrivers suck is nearly physically painful to watch.
"I don't know what I'm doing, but someone who knows what they're doing anecdotally solved a completely different problem using tool XYZ, so for lack of any better idea, lets copy them".
If you're familiar with cargo cult science there is an enormous miasma of cargo cult engineering fogging up the entire database arena not just nosql.
Another concept that needs to be in the discussion is the "no silver bullet" rule from programming applies to database design, like it or not. Can't just sprinkle magic nosql pixie dust on any old random problem and expect it to work, any more than applying any random programing fad to any random problem will work.
The (old and new) tools are actually pretty interesting, although often poorly engineered (by the end user) and implemented. Its the persistent anti-patterns and non-engineering design technique that I'm properly arrogant and condescending toward.
Hammers are a cool new invention and have some great unusual new applications, but they don't install deck screws any better than the legacy screwdriver. Laborers on the job randomly mixing screws nails screwdrivers and hammers on the job and then internet discussions about how hammers and/or screwdrivers suck is nearly physically painful to watch.
Regardless of whether you are right or not, your tone is very off-putting. This is not an academic concern – this sort of attitude is a huge part of why the tech community ends up being very exclusionary. Oh yes, watching someone make mistakes while learning how to build a web application or learn regular expressions is hilarious. Much better to laugh condescendingly while watching them struggle than try to help. Concocting "straw man" noobs everywhere who do things in obviously wrong ways doesn't make you more right. And it definitely doesn't make you smarter. It just makes you seem like a jerk and makes it a little harder for people to stomach getting into the tech field. Please stop.
The most popular corporate world database system at this time is Excel. And its not a very good DBMS, and the world is full of spreadsheet guys who know nothing about database theory, designing horrific abominations.
On a large scale I think the future will be interesting, will "real databases" remain in IT land or will it move out in the world?
Don't laugh, I'm just barely old enough that when I started out, printing, file management, physical media management, static network address configuration (not just IP in the old days), and backups were done solely by IT professionals in the machine room, and now for better or worse every noob does it (or tries to do it) themselves. If in the future someone comes up with software to automate away DBAs...
Or it might go the other way and the corporate business database of the future will be super turbo hyper i-Excel.
On a large scale I think the future will be interesting, will "real databases" remain in IT land or will it move out in the world?
Don't laugh, I'm just barely old enough that when I started out, printing, file management, physical media management, static network address configuration (not just IP in the old days), and backups were done solely by IT professionals in the machine room, and now for better or worse every noob does it (or tries to do it) themselves. If in the future someone comes up with software to automate away DBAs...
Or it might go the other way and the corporate business database of the future will be super turbo hyper i-Excel.
Anyone who cares about databases absolutely must check out Datomic. If you want to know the future of relational databases for the next 15 years, I think it is here. Practically infinite client side scaling, logic based querying, ACID, runs on several foundations, supports joins, all data is permanent so it stores the entire history of every fact. You can query "right now" or in the past at any known time. The database lives on your application stack the same way git checks out your code on a development machine, and you push and pull data to bring your application up to date with other stacks.
It doesn't meet every single need you might ever have, but it sure meets a whole lot more then any other database I've seen. http://www.datomic.com
It doesn't meet every single need you might ever have, but it sure meets a whole lot more then any other database I've seen. http://www.datomic.com
I'm really skeptical about Datomic. I think it has some great features, but I have a hard time believing that ACID transactions is one of them.
I've asked about this on their Google Group: https://groups.google.com/forum/#!topic/datomic/zkMV50VKw2s
As of right now, there are still questions left unanswered.
I've asked about this on their Google Group: https://groups.google.com/forum/#!topic/datomic/zkMV50VKw2s
As of right now, there are still questions left unanswered.
Also, "as of" reporting is a built-in feature requiring no additional configuration.
I still think its amazing that Facebook is running off MySQL. Such a tremendous amount of user data, placed in a system that many deem inferior.
I'm not saying they made a bad choice, I'm just impressed at how they've managed to make MySQL fit their needs.
I'm not saying they made a bad choice, I'm just impressed at how they've managed to make MySQL fit their needs.
I wonder if it fits their needs or they are stuck with it? I mean, at some point when you hit petabyte scales, is it even plausible to move to other data stores / rebuild indexes etc? I would not dare to do that.
I'm guessing the way they use MySQL is similar to the way they use PHP?
Impressive indeed.
Impressive indeed.
I was under the impression they use it more as a sharded "key-values" store?
ie. no joins - the part of the database that makes it relational.
ie. no joins - the part of the database that makes it relational.
I concur. In our shop we also use MySQL without the joins. That's like an efficient key-value store, but there must be a better solution somewhere: all the joins are done in client code, and caching too. That's a great amount of error-prone code that probably shouldn't be necessary.
"all the joins are done in client code"
What's the point of joining in the application level? If you're going to join, why not do it in the database? That should be both faster and more convenient (unless your schemas aren't relational at all, in which case I don't see why you'd use a relational database)
What's the point of joining in the application level? If you're going to join, why not do it in the database? That should be both faster and more convenient (unless your schemas aren't relational at all, in which case I don't see why you'd use a relational database)
Data for User A is on server 9, and data for User B is on server 43. Joins across servers is not really supported in mysql.
Another issue is connections between DB servers. Generally you want your DB server to be as fast as possible, so having it handle the connections to other servers slows everything down. If you offload the DB server connections to your web server, you can easily scale by just adding more web servers and having each DB server handle only its own data.
The best solution would be some type of 'mysql proxy' that could run on every web server that would transparently handle the joins between all the different mysql data servers. I think I saw a project attempting to do this awhile back, but didn't really keep track.
Another issue is connections between DB servers. Generally you want your DB server to be as fast as possible, so having it handle the connections to other servers slows everything down. If you offload the DB server connections to your web server, you can easily scale by just adding more web servers and having each DB server handle only its own data.
The best solution would be some type of 'mysql proxy' that could run on every web server that would transparently handle the joins between all the different mysql data servers. I think I saw a project attempting to do this awhile back, but didn't really keep track.
This is because join negatively impact the scalability of a DB:
- When huge amount of traffic hits your DB, out of the most commonly run-out resources - CPU, Memory, and IO - CPU is usually the one that runs out first. And Join uses quite a bit of CPU.
- When you have a lot of data and have to partition it, you will feel so lucky if you don't have to deal with distributed joins.
> that many deem inferior.
> If you bother to pay attention what FB DB engineers say you will know that they have their arms shoulder deep into the innards of the database itself, I/O stack, all that jazz.
Most of us can't afford that, and want something that's pretty good out of the box. I've used Mysql a number of times, but it always seems to have more gotchas for what I need to do with it than Mysql.
Most of us can't afford that, and want something that's pretty good out of the box. I've used Mysql a number of times, but it always seems to have more gotchas for what I need to do with it than Mysql.
It's just going to be different dbs in the backend working with what they do best in tandem.
Lucence base for search. RMDB for money/transactions. And other noSQL type for just data that isn't critical, don't care much about relation, and just need it to be fast.
Of course, perhaps PostgreSQL and other RMDB can have noSQL attributes then noSQL would have some good competition.
PostgreSQL if they can make it easier to clustered, like cassandra or elasticsearch. And get a better fuzzy string and other text search capability in there then now we're talking. But I guess it's just me dreaming.
Lucence base for search. RMDB for money/transactions. And other noSQL type for just data that isn't critical, don't care much about relation, and just need it to be fast.
Of course, perhaps PostgreSQL and other RMDB can have noSQL attributes then noSQL would have some good competition.
PostgreSQL if they can make it easier to clustered, like cassandra or elasticsearch. And get a better fuzzy string and other text search capability in there then now we're talking. But I guess it's just me dreaming.
By the sounds of it you don't understand how banks work today. They are eventually consistent and would work great with databases like Cassandra. And the growth of NoSQL today is in enterprises which are almost entirely mission critical.
It's actually the smaller SQL databases (MySQL, PostgreSQL) that are being used for non mission critical uses.
It's actually the smaller SQL databases (MySQL, PostgreSQL) that are being used for non mission critical uses.
postgresql XC is master-master, shardable, cross-shard transactions, data partitioning across nodes, distributed queries
Yes, but it's side project so it doesn't get such attention\confidence as postgresql. But I hope the project won't get abandoned and will evolve into something very powerful.
It's also not part of the mainline code base which gives it the impression of being unsupported and untrustworthy.
"Obamacare, for better or for worse, is being launched on a NoSQL database system" - silver bullet did not help (along with HTML5 etc. whatever).
So are you implying that a regular DB would have fared better?
I think it was a failure because of the complex law background and interfacing with all kinds of legacy systems it has to draw data from.
I think it was a failure because of the complex law background and interfacing with all kinds of legacy systems it has to draw data from.
Obamacare website failed because of bad capacity planning.
And from reports coming out it was the Oracle Identity system that was causing many of the headaches.
And from reports coming out it was the Oracle Identity system that was causing many of the headaches.
Could it be that facebook complete lack of innovation is due to them being stuck for years with the same issues on the db ?
This is an example of an interesting problem with "innovative" databases which is vaguely like the OSI networking model
The first layer is business need, without one there is no point. You need to start here. Facebook is empty here, so no need for innovation and no concrete ideas about what to innovate about, means no point discussing arch or later tech.
The next layer is architecture. Do you need joins? Well then use them. Do you need transactions, or not? Basically a big key value store?
The final step / layer is tech. You can do a boring key value store on mysql just fine. There might be a slightly faster or cheaper software choice. Or maybe not. Or maybe some nosql thing fits the architecture.
This is a huge problem I see with "the whole market is going to be disrupted totally and only I know how" type articles. So I herd cats around a quasi-geographic database at work (not directly geographic, but related). The only way to change its scale or business needs is to modify the geography of the United States or dramatic demographic changes that would even theoretically take generations to breed the eventual customers. So layer one can't change for that job.
The arch doesn't seem like it can be improved very much. So layer two can't change for that job.
Therefore because layer one and layer two are static, layer three will utterly change completely. Because he says so. LOL
The first layer is business need, without one there is no point. You need to start here. Facebook is empty here, so no need for innovation and no concrete ideas about what to innovate about, means no point discussing arch or later tech.
The next layer is architecture. Do you need joins? Well then use them. Do you need transactions, or not? Basically a big key value store?
The final step / layer is tech. You can do a boring key value store on mysql just fine. There might be a slightly faster or cheaper software choice. Or maybe not. Or maybe some nosql thing fits the architecture.
This is a huge problem I see with "the whole market is going to be disrupted totally and only I know how" type articles. So I herd cats around a quasi-geographic database at work (not directly geographic, but related). The only way to change its scale or business needs is to modify the geography of the United States or dramatic demographic changes that would even theoretically take generations to breed the eventual customers. So layer one can't change for that job.
The arch doesn't seem like it can be improved very much. So layer two can't change for that job.
Therefore because layer one and layer two are static, layer three will utterly change completely. Because he says so. LOL
Best I recollect, FB is quite innovative in the data center.
Also, it would be nice of you to step in and contribute to an open source database engine if your expertise is such that you can dismiss what FB did with them as lacking innovation. I'd wager they'll be all ears open for your explanation how to do multimaster replication and distributed joins efficiently while remaining ACID compliant.
Also, it would be nice of you to step in and contribute to an open source database engine if your expertise is such that you can dismiss what FB did with them as lacking innovation. I'd wager they'll be all ears open for your explanation how to do multimaster replication and distributed joins efficiently while remaining ACID compliant.
I wasn't talking about facebook technical innovations, but rather end-user new features, sorry i wasn't clear.
aren't they using many other dbs for different stuff, like hbase for messagess, some other for graphing etc
I don't know. That was an honest question. The original post seemed to mention facebook being stuck on mysql as a big issue. So that made me wonder if that could be a reason why the only useful feature facebook released was facebook chat, and it was more than two years ago.
You don't really understand how popular websites work do you ?
Their users don't want innovation. They want a stable platform with moderate and infrequent changes.
Their users don't want innovation. They want a stable platform with moderate and infrequent changes.
By the way, as long as we're speaking about DB future, did anyone try foundationdb? Does it live up to expectations?
What about H2?
http://www.h2database.com/html/main.html
I can't think of a better reason NOT to use it. Can someone who knows a lot about db please comment on this.
http://www.h2database.com/html/main.html
I can't think of a better reason NOT to use it. Can someone who knows a lot about db please comment on this.
https://en.wikipedia.org/wiki/The_Third_Manifesto
The difficult problems since that time turned out to be scalability and parallelism, neither of which were addressed by The Third Manifesto. So instead of D, we got Map Reduce and NoSQL.
Who knows what the hard problems of the next 15 years will be.