How and why to make your software faster(incoherency.co.uk)
incoherency.co.uk
How and why to make your software faster
http://incoherency.co.uk/blog/stories/how-to-make-software-faster.html
10 comments
And toss in a bloom filter to test for membership and reduce the likelihood of even needing to call the db.
But that also loses your backpressure - increasing chances that sooner or later (depending on platform/technology), the whole server will go down in a bad way.
> If I can't make your worst-performing page load 50% faster, you don't have to pay the bill.
Interesting approach. This is also nicely recursive, with each run being more expensive. I would prefer that statement to have a cost cap. Say "... in three days of work".
Interesting approach. This is also nicely recursive, with each run being more expensive. I would prefer that statement to have a cost cap. Say "... in three days of work".
Sure, I would work out something mutually agreeable before saying "tadaa! That's 9 grand please!" :)
Thanks for reading.
Thanks for reading.
Just buy a bigger box with double the memory?
Fun things I discovered in just my last performance tuning engagement:
* Database with homegrown software load-balancer in front of it, and the database replicated to a half-dozen subscribers. Load balancing required database calls, and the subscribers had their LUNs on the same array as the main database. So 7 different servers, but all beating the same storage. In the end, we ended up with no load balancing (just the single instance).
* Application server with lots of agents (one on every workstation). Agents were checking in every 200ms, even if they had nothing to report. Result was lots of tiny database transactions (for logging no activity). Throttled that down to 10 seconds (product managers agreeing that was the more realistic threshold) and batching the updates, made the logging asynchronous, and saw the database transaction log no longer waiting on log writes constantly.
* Bad clustering keys for tables. Some tables that were being searched constantly as heaps. Sharded tables with divergent schemas. This was fun, as had to conform everything while it was live (got agreement on what was reasonable delay for synch jobs). Chose correct clustering keys, compressed the tables, and then got rid of other indexes that were no longer necessary. Cut IO by 99%, and writes were now just appends.
* Badly written queries using inadequate indexes on badly structured tables. TL;DR: An analysis process took 24 hours, and was tuned to take 4.5 hours. I spent a day on it to cut it down to an hour while I worked on other things. We then spent 2 weeks refactoring it to cut it down to about 8 minutes, despite the underlying data model being bleh. We also added ad-hoc functionality to do stuff out-of-band of the regular scheduled job, with an average response time of 6ms.
* ORM lazy loading by default, including for things that constantly retrieved collections. Rewrote them to be programmatically eager loaded and reduced database calls from millions a day to thousands a day.
* And a little bit of hardware. SAN HBA links were moved form dual 1Gbps to dual 10Gbps. Some SSDs added. Ended up reducing IO latency by 96-98%, depending on the server. Needed to happen, as IO latency was in hundreds of ms, and we were able to get it down to where it should be (< 10ms for reads, 1-3ms for writes).
Non-scaling SAAS stack went from falling on its face with an entire rack of hardware, to being cut down to 1/3rd of a rack of hardware and the hardware basically twiddling its thumbs. The organization was in the process of building of entirely new SAAS platform in a completely different tech stack -- they immediately started questioning that decision when the old stack handled everything trivially, and they knew there was a lot more tuning that still could be done.
* Database with homegrown software load-balancer in front of it, and the database replicated to a half-dozen subscribers. Load balancing required database calls, and the subscribers had their LUNs on the same array as the main database. So 7 different servers, but all beating the same storage. In the end, we ended up with no load balancing (just the single instance).
* Application server with lots of agents (one on every workstation). Agents were checking in every 200ms, even if they had nothing to report. Result was lots of tiny database transactions (for logging no activity). Throttled that down to 10 seconds (product managers agreeing that was the more realistic threshold) and batching the updates, made the logging asynchronous, and saw the database transaction log no longer waiting on log writes constantly.
* Bad clustering keys for tables. Some tables that were being searched constantly as heaps. Sharded tables with divergent schemas. This was fun, as had to conform everything while it was live (got agreement on what was reasonable delay for synch jobs). Chose correct clustering keys, compressed the tables, and then got rid of other indexes that were no longer necessary. Cut IO by 99%, and writes were now just appends.
* Badly written queries using inadequate indexes on badly structured tables. TL;DR: An analysis process took 24 hours, and was tuned to take 4.5 hours. I spent a day on it to cut it down to an hour while I worked on other things. We then spent 2 weeks refactoring it to cut it down to about 8 minutes, despite the underlying data model being bleh. We also added ad-hoc functionality to do stuff out-of-band of the regular scheduled job, with an average response time of 6ms.
* ORM lazy loading by default, including for things that constantly retrieved collections. Rewrote them to be programmatically eager loaded and reduced database calls from millions a day to thousands a day.
* And a little bit of hardware. SAN HBA links were moved form dual 1Gbps to dual 10Gbps. Some SSDs added. Ended up reducing IO latency by 96-98%, depending on the server. Needed to happen, as IO latency was in hundreds of ms, and we were able to get it down to where it should be (< 10ms for reads, 1-3ms for writes).
Non-scaling SAAS stack went from falling on its face with an entire rack of hardware, to being cut down to 1/3rd of a rack of hardware and the hardware basically twiddling its thumbs. The organization was in the process of building of entirely new SAAS platform in a completely different tech stack -- they immediately started questioning that decision when the old stack handled everything trivially, and they knew there was a lot more tuning that still could be done.
That's the easy way out - therefore, step 0. Step 1 is "we tried throwing hardware at it, didn't help a lot, now what?" ("Well throw more hardware then" rarely gives a measurable improvement.)
To the downvoter: Maybe in your fantasy world, throwing more computing power at problems solves Eeeverything, including P ?= NP. I do wonder - why people even bother offering such services as the OP, if it's so pointless?
To the downvoter: Maybe in your fantasy world, throwing more computing power at problems solves Eeeverything, including P ?= NP. I do wonder - why people even bother offering such services as the OP, if it's so pointless?
I think you may have been downvoted because the article just explains: step 0 should always be profiling and then deciding what you need, either more hardware or better code.
I once spent weeks optimizing code on a "best in the world" (for my branch of work) open source project because it made the assumption that it'd run on SSDs, hence it made zillions of small reads from the disk. There were two approaches to solve that: buy an SSD, or refactor the code. But it was through profiling that I found the answer (and in my economic situation at the time, buying an SSD was definitely not viable, much less a step 0!).
I once spent weeks optimizing code on a "best in the world" (for my branch of work) open source project because it made the assumption that it'd run on SSDs, hence it made zillions of small reads from the disk. There were two approaches to solve that: buy an SSD, or refactor the code. But it was through profiling that I found the answer (and in my economic situation at the time, buying an SSD was definitely not viable, much less a step 0!).
Unless the box is heavily overloaded, that won't help. Profile it and see what needs fixing!
Sometimes that doesn't help.
At work we were debugging why a process that is usually quite fast (< 5min) was taking close to 2 hours once we got to the load we predict for this coming fall.
We noticed the box it was running on was running out of memory, so we assumed resizing it to have ~4x the RAM would help, but surprisingly it didn't. The actual answer turned out to be algorithmic, and we were able to rewrite it to use an O(n) algorithm rather than an O(n^2) algorithm.
At work we were debugging why a process that is usually quite fast (< 5min) was taking close to 2 hours once we got to the load we predict for this coming fall.
We noticed the box it was running on was running out of memory, so we assumed resizing it to have ~4x the RAM would help, but surprisingly it didn't. The actual answer turned out to be algorithmic, and we were able to rewrite it to use an O(n) algorithm rather than an O(n^2) algorithm.
Some companies box is a warehouse with rows of racks of servers consuming hundreds of kilowatts, if not a megawatt. Those are exceptional cases, but it helps to point out how "just buy a bigger box" eventually has limits.
>Just buy a bigger box with double the memory?
For what it's worth, a quite staggering number of sites I've fixed up have had databases with varchar ID columns, and no indexes anywhere.
Thankfully the DB sizes have been small enough that fixing this has taken a few seconds to a few minutes max of DB downtime, with instant, seriously-impressive-looking results afterwards.
For what it's worth, a quite staggering number of sites I've fixed up have had databases with varchar ID columns, and no indexes anywhere.
Thankfully the DB sizes have been small enough that fixing this has taken a few seconds to a few minutes max of DB downtime, with instant, seriously-impressive-looking results afterwards.
I wonder when this type of work will go away. I see it everywhere.
it will never. databases are hard and people are idiots. nevermind something as hard as SQL optimization and index design -- there are LOTS of people out there who think creating a new logical database for every new customer is a good idea, resulting in 2000+ databases on a system when there should be 4. people who think HA is a waste of money because nothing has ever failed on them. cheapskates who will under-spec deployments and then yell at you when it doesn't fit their data. people are stupid. do not underestimate the stupidity of your average tech industry "professional". this is why i think there should be a traditional engineering certification process for this stuff.
also, i simply don't buy the assertion that any ops or dev can be automated in any meaningful way. the only thing it's doing is making it easier for morons to shoot themselves in the foot by ignoring the experts and then run to the same experts like a little child when everything catches on fire. see: recent story of a guy who deleted 1500 systems at once. oops.
are we all doing more, or less work than 5 years ago? has all this cloud and deployment tech made any less work for anyone, or destroyed any jobs? lol no. i repeat emphatically LOL NO. such a claim is just absurd in my view. anyone who claims such a nonsensical statement has never done actual ops work where you get called if shit goes south.
the smartest thing amazon ever did was simply give people a button to push to give them more money when they're faced with their own overwhelming stupidity. ain't nobody to blame but yo'self then.
also, i simply don't buy the assertion that any ops or dev can be automated in any meaningful way. the only thing it's doing is making it easier for morons to shoot themselves in the foot by ignoring the experts and then run to the same experts like a little child when everything catches on fire. see: recent story of a guy who deleted 1500 systems at once. oops.
are we all doing more, or less work than 5 years ago? has all this cloud and deployment tech made any less work for anyone, or destroyed any jobs? lol no. i repeat emphatically LOL NO. such a claim is just absurd in my view. anyone who claims such a nonsensical statement has never done actual ops work where you get called if shit goes south.
the smartest thing amazon ever did was simply give people a button to push to give them more money when they're faced with their own overwhelming stupidity. ain't nobody to blame but yo'self then.
On the flip side, learning resources will continue to improve. Now days all one would have to do is google "database optimizations" and immediately see things like indices etc. Furthermore if they're using a framework, which most are, the search becomes "how to speed up django" which leads to a lovely documented page hand holding you through whatever you need to do.
> there are LOTS of people out there who think creating a new logical database for every new customer is a good idea, resulting in 2000+ databases on a system when there should be 4.
I'm genuinely curious as to why you feel this way?
I'm genuinely curious as to why you feel this way?
An ungodly amount of reasons. The ability to manage that many, for one. Those poor devops folk who have to deal with that.
On a more technical side as an example, the number of open files can only be so high. You really start to hit the limit when you do things like that, which is when the most interesting kinds application crashes happen. "Just increase the file count." results in "Our report shows that an OldServer did not include a necessary open file limit config change. Engineer #582 has been fired."
On a more technical side as an example, the number of open files can only be so high. You really start to hit the limit when you do things like that, which is when the most interesting kinds application crashes happen. "Just increase the file count." results in "Our report shows that an OldServer did not include a necessary open file limit config change. Engineer #582 has been fired."
CLI tooling has solved a lot of the problems that arise from many. Scripts for migration, host transfer, creation and deletion took some time to build out, but do work fine.
I suppose we are lucky to have a very consistent workload per logical database.
This allowed us to calculate our costs ahead of time. We found that we could keep DB costs at ~$1/customer/month. For a B2B SaaS product we were satisfied with that cost.
I suppose we are lucky to have a very consistent workload per logical database.
This allowed us to calculate our costs ahead of time. We found that we could keep DB costs at ~$1/customer/month. For a B2B SaaS product we were satisfied with that cost.
My personal preference for a single multi-tenanted schema as opposed to 2000+ schemas is:
Unless you require per customer customisations (which is a fast way to ensure you have an upper limit on the number of customers you can support) there is no need to duplicate. Use a single schema where every table with per-customer data has a tenant id column.
Most decent RDBMS have a good way to ensure isolated locks across tenant ids e.g. MySQL partitions.
Most decent RDBMS have a good way to ensure isolated locks across tenant ids e.g. MySQL partitions.
Understood.
Currently, I manage several hundred logical databases for a B2B SaaS product. This has the potential to grow to several thousand over the course of years.
My customers enjoy that (arbitrary) piece of mind that their data is logically separated from all else. I enjoy not having to remember to include a tenant_id clause.
Our traffic pattern is quite steady per hour of day and per customer. That, plus using RDS for horizontal scaling (fixed # of logical DBs per host) allowed us to take this route.
I suppose the decision is a matter of circumstance and preference, but I don't think this is necessarily "wrong" to do.
Currently, I manage several hundred logical databases for a B2B SaaS product. This has the potential to grow to several thousand over the course of years.
My customers enjoy that (arbitrary) piece of mind that their data is logically separated from all else. I enjoy not having to remember to include a tenant_id clause.
Our traffic pattern is quite steady per hour of day and per customer. That, plus using RDS for horizontal scaling (fixed # of logical DBs per host) allowed us to take this route.
I suppose the decision is a matter of circumstance and preference, but I don't think this is necessarily "wrong" to do.
it's going to work fine until it doesn't. that's not going to be a good day. smart people can rationalize anything.
see? LOTS.
When companies figure out that hiring "one guy that can do everything by himself!" is a bad idea. So...... not likely anytime soon.
Except... sometimes one guy really can do everything by himself, and do it much better than the other folks. However, usually the person who can do that is "too expensive" and they'd rather spread the cost (or a higher cost) over 3-4 people.
It's not just talent/skills - there's more communication overhead when dealing with a team, and when the team has a range of skill levels on it, the comm overhead can make things worse (if not managed well)
It's not just talent/skills - there's more communication overhead when dealing with a team, and when the team has a range of skill levels on it, the comm overhead can make things worse (if not managed well)
I have seen the one guy replaced by a team of 5. The team got less done than the guy.
I have a longstanding suspicion that developer cost climbs slower than developer skill and productivity, meaning companies should always hire the best they can get, because they'll maximize productivity that way.
I have a longstanding suspicion that developer cost climbs slower than developer skill and productivity, meaning companies should always hire the best they can get, because they'll maximize productivity that way.
IME, it climbs WAY slower.
I am the one guy fixing these things all by myself.
Hah. Why think when you can spend money?
Firstly, this doesn't help if you're CPU-bound or write-bandwidth-bound, both of which are quite common.
Secondly, you may not be able to simply get a bigger box. Most motherboards have effective RAM limits. Bigger iron gets exponentially more expensive. Jeff Atwood on this: http://blog.codinghorror.com/scaling-up-vs-scaling-out-hidde...
(I remember about a decade ago when our chip design startup bought a 128GB RAM box because that was the largest available at sane pricing ...)
Firstly, this doesn't help if you're CPU-bound or write-bandwidth-bound, both of which are quite common.
Secondly, you may not be able to simply get a bigger box. Most motherboards have effective RAM limits. Bigger iron gets exponentially more expensive. Jeff Atwood on this: http://blog.codinghorror.com/scaling-up-vs-scaling-out-hidde...
(I remember about a decade ago when our chip design startup bought a 128GB RAM box because that was the largest available at sane pricing ...)
Not calling this dishonest, because it's a fine way to give a guarantee to people who'd don't think much about performance. However, it's one of those claims that I suspect sounds more impressive than it is, depending on how you read it. Unless your system is stupidly simple or has had significant attention paid to performance, there's a good chance your worst page is spending most of its time in some awful O(n^2) algorithm, and will be an easy fix.
The world is full of things that easy to fix if you have the experience, but remain unfixed.
> "In the 1880s, James MacNeill Whistler, as plaintiff in a libel action, was challenged, "For two days' labour, you ask two hundred guineas?" "No, I ask it for the experience of a lifetime." That seems an apt summary of the message of this legend."
http://www.snopes.com/business/genius/where.asp
> "In the 1880s, James MacNeill Whistler, as plaintiff in a libel action, was challenged, "For two days' labour, you ask two hundred guineas?" "No, I ask it for the experience of a lifetime." That seems an apt summary of the message of this legend."
http://www.snopes.com/business/genius/where.asp
"The key to making programs fast is to make them do practically nothing."
I believe the quote original is here[0]
Maybe: s/practically nothing/less
[0]https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
Maybe: s/practically nothing/less
[0]https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
How to make things go faster: Do fewer things.
"no code is faster than no code"
[0]: https://twitter.com/davecheney/status/715421274722271232
"no code is faster than no code"
[0]: https://twitter.com/davecheney/status/715421274722271232
>> "If I can't make your worst-performing page load 50% faster, you don't have to pay the bill."
Unlikely that the worst performing page is the most valuable page; which is to say that offers should focus on providing value.
Unlikely that the worst performing page is the most valuable page; which is to say that offers should focus on providing value.
Reminds me of one of my very first jobs where I was to optimize some MATLAB code. I found a function that looked obviously inefficient and after a days work I'd managed to get a ~10-20x speedup on that function. I was super proud of myself until it dawned on me that all I'd really achieved was to knock 30-60 seconds of a 6-8 hours process.
Also, I wouldn't necessarily limit yourself to a single page. Lots of companies have offline processes that are painfully slow, and are very good candidates for optimization.
I once worked at a company where I noticed an employee would come in over an hour earlier than normal on Monday mornings. It turns out she got in early to run a report that the CEO liked to see every Monday morning. An hour or two later I added an index, and refactored some SQL so that we could do more in a single query rather than N queries, and the report time went down from 45 minutes to 3 minutes.
I once worked at a company where I noticed an employee would come in over an hour earlier than normal on Monday mornings. It turns out she got in early to run a report that the CEO liked to see every Monday morning. An hour or two later I added an index, and refactored some SQL so that we could do more in a single query rather than N queries, and the report time went down from 45 minutes to 3 minutes.
Fair point.
A page that takes 10s to load that you never look at is a much worse candidate than one that takes 8s to load and you look at 20 times a day.
A page that takes 10s to load that you never look at is a much worse candidate than one that takes 8s to load and you look at 20 times a day.
I'm sorry, but if I had to look at a page 20 times a day that took 8 seconds to load, I would help that web app commit suicide.
There is absolutely no call for a web app that badly written to exist in 2016. Period.
There is absolutely no call for a web app that badly written to exist in 2016. Period.
OK, that's fair.
So how about this: You go over there (points to corner) and make angry and dogmatic proclamations of what should and shouldn't be, while the rest of us go out into the world, where these apps do exist in large numbers and see heavy usage supporting very valuable workflows, and collect obscene salaries making them better.
So how about this: You go over there (points to corner) and make angry and dogmatic proclamations of what should and shouldn't be, while the rest of us go out into the world, where these apps do exist in large numbers and see heavy usage supporting very valuable workflows, and collect obscene salaries making them better.
No piece of code is written perfectly the first time. A software application (web or otherwise) tends to be a bundle of validations and compromises when written under the constraints of budgets, deadlines, and priorities.
There are many scenarios where a page used 20 times a day takes 8 seconds to load. For example, an admin dashboard page that shows some statistics on some aspect of customer usage written early on in a startup's life might only need to be looked at once a month by one of the founders (since it app has little traction, they have bigger things to worry about than seeing statistics from an n of 3).
So, it's hammered out with inefficient queries, improper indexes, etc. (after all, we don't even have a clear enough view of actual usage yet to know what indexes or queries we should really be doing).
Time goes on, the startup picks up traction, and before you know it, it's April 15, 2016, and the application handles millions interactions a week across its users. Someone on the support team at some point in the near past realized that they should be looking at these usage statistics to help drive improvements to the interface of the application, but the page now takes 8 seconds to load, because of all the data stored in that table, and because of the inefficient queries, improper indexes, etc.
Due to the nature of the business and the aforementioned constraints of budgets, deadlines, and priorities, this person has spent the last month or so just dealing with the 8-second page load. But now, it's a priority, and someone can go in, look at the data that's actually there, and figure out the best way to refactor, optimize, and improve it.
The point is, there are many scenarios where a web app can have a page checked 20 times a day that takes 8 seconds to load without assuming it's so badly written, and it certainly has a reason to exist (after all, it is partially the success of the app and value it provides to people that has caused this issue in the first place), and it doesn't need to die, it needs to adapt.
There are many scenarios where a page used 20 times a day takes 8 seconds to load. For example, an admin dashboard page that shows some statistics on some aspect of customer usage written early on in a startup's life might only need to be looked at once a month by one of the founders (since it app has little traction, they have bigger things to worry about than seeing statistics from an n of 3).
So, it's hammered out with inefficient queries, improper indexes, etc. (after all, we don't even have a clear enough view of actual usage yet to know what indexes or queries we should really be doing).
Time goes on, the startup picks up traction, and before you know it, it's April 15, 2016, and the application handles millions interactions a week across its users. Someone on the support team at some point in the near past realized that they should be looking at these usage statistics to help drive improvements to the interface of the application, but the page now takes 8 seconds to load, because of all the data stored in that table, and because of the inefficient queries, improper indexes, etc.
Due to the nature of the business and the aforementioned constraints of budgets, deadlines, and priorities, this person has spent the last month or so just dealing with the 8-second page load. But now, it's a priority, and someone can go in, look at the data that's actually there, and figure out the best way to refactor, optimize, and improve it.
The point is, there are many scenarios where a web app can have a page checked 20 times a day that takes 8 seconds to load without assuming it's so badly written, and it certainly has a reason to exist (after all, it is partially the success of the app and value it provides to people that has caused this issue in the first place), and it doesn't need to die, it needs to adapt.
All the way back to 2015, I got a job that said "we have a page (in a Windows app) that loads in 5 hours; we'd like that to be much faster, preferably in under a minute".
I got it down to a few seconds, but that's not the important part: that was an application that was being used, daily. That page would just take 5 hours to load.
I got it down to a few seconds, but that's not the important part: that was an application that was being used, daily. That page would just take 5 hours to load.
I used to work at a place where the front-end guys had a big push to get the page load times down below a minute. People paid tens of thousands of dollars for a subscription to it. The alternative was that you pay another company a couple million and wait 6 months for a binder to arrive in the mail, so the customers were content with the page load times.
The backend team was more worried about keeping daily processing and offline reports below 24 hours. The offline reports were essentially SQL queries.
The backend team was more worried about keeping daily processing and offline reports below 24 hours. The offline reports were essentially SQL queries.
There's actually a really good reason: As data gets bigger, so do page speeds. Take a look at Google Analytics. Their "slow result" takes quite a bit to process.
If you're just rendering a couple database fields to a page? Sure, that's easy
If you're just rendering a couple database fields to a page? Sure, that's easy
He's not saying that he won't work to improve the performance of the entire application. He's merely limiting the conditions under which he won't get paid to the worst performing page.
Nice post, but why call out C and Perl when talking about web apps? Maybe highlight some profilers for some more common web stacks.
For example, I'm currently writing a book on web app performance (focusing on ASP.NET Core) and apart from the stuff built into Visual Studio there are also tools such as Glimpse[0] and Prefix[1].
[0] http://getglimpse.com
[1] http://www.prefix.io
For example, I'm currently writing a book on web app performance (focusing on ASP.NET Core) and apart from the stuff built into Visual Studio there are also tools such as Glimpse[0] and Prefix[1].
[0] http://getglimpse.com
[1] http://www.prefix.io
And for PHP, XHProf and Blackfire (among others).
For Golang, its pretty much in std lib: https://golang.org/pkg/net/http/pprof/
What's your thought on mongodb - i've recently started a project that is running in prod now for a customer and I spent almost two weeks deciding between mysql and mongodb. Ultimately went with mongodb but it's not my forte. And I've been thinking that at some point the calls to the database are going to slow down. And i have no idea where to start looking to improve mongodb performance. (MEAN stack application) I'm sure I'll be doing a bit of research soon on that once i get the base feature set working.
I've never used mongo in anger, but that doesn't matter, my answer is:
Profile the application. Find out what it's spending too much time on, and figure out how to make it do less of that.
Profile the application. Find out what it's spending too much time on, and figure out how to make it do less of that.
Can't upvote this enough. You cannot reliably improve what you cannot reliably measure, and Amdahl's Law will tell you where to work.
I wish you every success in this business - if it takes off, I might try it myself. I found `oprofile` to be my preferred Linux profiler, and haven't worked out what the corresponding Windows solution is yet.
I wish you every success in this business - if it takes off, I might try it myself. I found `oprofile` to be my preferred Linux profiler, and haven't worked out what the corresponding Windows solution is yet.
I would argue that there is literally never a reason to use MongoDB over other solutions. It looks tempting at first, but in reality it has all the downsides of any other database and a whole lot more junk heaped on top.
YMMV of course; that might just be my biased view after weeks of trying to improve performance.
YMMV of course; that might just be my biased view after weeks of trying to improve performance.
What kind of research did you do where you ended up actually choosing mongodb? Serious question.
Both MongoDB and MySQL were started by people who had no background in writing databases. They were blazingly fast compared to existing RDBMS, but once they started adding features that actually provided things that people assumed a database should have (primarily in area of consistency) they no longer look that great.
What makes MySQL better than MongoDB is that is older and closer to what a database should do. They still have their warts and some of the warts might not be removed, since it would break compatibility.
At this point MongoDB does not have anything going for them. Its key benefit was their speed, but as it turned out that was because data was stored mostly in RAM if Mongo crashed or there was a power loss then you most likely would lose significant (possibly all) of your data. Since then they fixed that and make it more reliable, it is worse in performance than a relational database [1] and it also doesn't scale well [2]
Essentially NoSQL databases were designed to be simple and without relational features in exchange for scalability and performance. Mongo you get neither. Mongo doesn't even try to benefit from the CAP theorem. It's neither always consistent nor always available [3].
You generally should always use a relational database, because in majority of cases you do have some schema and you expect data to be consistent. NoSQL databases (especially ones that are AP in CAP) are generally good for specialized use cases, things that have no relations and are acceptable to be wrong or missing occasionally. For example storing logs, or user sessions etc.
Lastly, regarding question about performance. You need to understand your data and what you are doing. At my previous job there was a database called region. It was intended for a task such as looking up latitude/longitude -> ZIP code, and also IP -> ZIP code.
That database was (and probably still is) running on 3 beefy machines running Mongo and contained data was about 13GB. One time I wanted to see how it would work in a relational database. So I loaded the dataset to a Postgres database and installed PostGIS and ip4r extensions. And you know what? The same data took only 600MB there and all queries took sub millisecond on smallest VM.
How come? Mongo does not understand IP addresses, so what they did is they converted an IP into a number and stored it as a 64bit integer in Mongo. Postgres on the other hand was simply storing IP ranges, and with an GiST index.
Why I'm telling you this? I think it is important to know that RDBMS databases have been used for a long time, and many problems were already solved there if you're having some performance problem chances are someone else did have it as well before you (in this case someone wrote an extension providing a new data type)
[1] http://www.enterprisedb.com/postgres-plus-edb-blog/marc-lins... [2] http://www.datastax.com/wp-content/themes/datastax-2014-08/f... [3] https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
What makes MySQL better than MongoDB is that is older and closer to what a database should do. They still have their warts and some of the warts might not be removed, since it would break compatibility.
At this point MongoDB does not have anything going for them. Its key benefit was their speed, but as it turned out that was because data was stored mostly in RAM if Mongo crashed or there was a power loss then you most likely would lose significant (possibly all) of your data. Since then they fixed that and make it more reliable, it is worse in performance than a relational database [1] and it also doesn't scale well [2]
Essentially NoSQL databases were designed to be simple and without relational features in exchange for scalability and performance. Mongo you get neither. Mongo doesn't even try to benefit from the CAP theorem. It's neither always consistent nor always available [3].
You generally should always use a relational database, because in majority of cases you do have some schema and you expect data to be consistent. NoSQL databases (especially ones that are AP in CAP) are generally good for specialized use cases, things that have no relations and are acceptable to be wrong or missing occasionally. For example storing logs, or user sessions etc.
Lastly, regarding question about performance. You need to understand your data and what you are doing. At my previous job there was a database called region. It was intended for a task such as looking up latitude/longitude -> ZIP code, and also IP -> ZIP code.
That database was (and probably still is) running on 3 beefy machines running Mongo and contained data was about 13GB. One time I wanted to see how it would work in a relational database. So I loaded the dataset to a Postgres database and installed PostGIS and ip4r extensions. And you know what? The same data took only 600MB there and all queries took sub millisecond on smallest VM.
How come? Mongo does not understand IP addresses, so what they did is they converted an IP into a number and stored it as a 64bit integer in Mongo. Postgres on the other hand was simply storing IP ranges, and with an GiST index.
Why I'm telling you this? I think it is important to know that RDBMS databases have been used for a long time, and many problems were already solved there if you're having some performance problem chances are someone else did have it as well before you (in this case someone wrote an extension providing a new data type)
[1] http://www.enterprisedb.com/postgres-plus-edb-blog/marc-lins... [2] http://www.datastax.com/wp-content/themes/datastax-2014-08/f... [3] https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
Do you offer https://en.wikipedia.org/wiki/Code_refactoring
Sure, email address in my profile.
Can someone ELI5 what profiling data is?
Here's an example of the result of profiling a a particular page view of a web application: http://moritz.faui2k3.org/tmp/otrs-nytprof/agentticketzoom-n...
The graph at the top shows you which methods took how much time. The wider a bar, the longer it took. So it's immediately obvious that there are two substitutions called from Kernel::Output::HTML::Layout::Output that take up nearly all of the time. That's where you can focus your optimizations on. The other methods only take so much time because they include calls to the offending methods.
You can click on the bars (or the the routines listed below the graph) to get a few of the code, annotated with call counts and time spent on each line.
This is from OTRS, when it still used its home-grown, hacky template system, which is the major offender here. After switching to a proper template system, this is what the same request looks like: http://moritz.faui2k3.org/tmp/otrs-nytprof/agentticketzoom-u...
You can see from the absolute numbers at the top that it's much faster (18.3s -> 2.12s), and there's no single spot anymore where all the time is spent.
It is rather typical that unoptimized code has one or more big time sinks, and profiling makes that obvious.
The graph at the top shows you which methods took how much time. The wider a bar, the longer it took. So it's immediately obvious that there are two substitutions called from Kernel::Output::HTML::Layout::Output that take up nearly all of the time. That's where you can focus your optimizations on. The other methods only take so much time because they include calls to the offending methods.
You can click on the bars (or the the routines listed below the graph) to get a few of the code, annotated with call counts and time spent on each line.
This is from OTRS, when it still used its home-grown, hacky template system, which is the major offender here. After switching to a proper template system, this is what the same request looks like: http://moritz.faui2k3.org/tmp/otrs-nytprof/agentticketzoom-u...
You can see from the absolute numbers at the top that it's much faster (18.3s -> 2.12s), and there's no single spot anymore where all the time is spent.
It is rather typical that unoptimized code has one or more big time sinks, and profiling makes that obvious.
A common technique for looking at a program (not a database query) is "stack sampling". A timer goes off N times a second, and records the call stack of the process/threads. Then, statistics are gathered from the set of call stacks.
E.g. - "60% of the time is in the sequence main() -> process_unit() -> validate_input() and the things called from there" or "45% of the time is in all of the call sequences which then lead back into write_line()" or things like that. Usually, you see patterns of a small number of slow functions that everything depends upon and/or, arranging the call stacks into a sort of "tree" of calls, a branch of the tree that you spend an inordinate amount of time in.
There are newer tools that provide graphical representation of this data as well (e.g. - show the tree as sort of a topographical map, with hotspots as peaks, and the call tree as geological strata)
Of course, if your program has a giganto routine that it never leaves, you might not learn much -- "do_all_the_work_inline() is executing 95% of the time!", unless you start looking at things at the line number level. Blech.
E.g. - "60% of the time is in the sequence main() -> process_unit() -> validate_input() and the things called from there" or "45% of the time is in all of the call sequences which then lead back into write_line()" or things like that. Usually, you see patterns of a small number of slow functions that everything depends upon and/or, arranging the call stacks into a sort of "tree" of calls, a branch of the tree that you spend an inordinate amount of time in.
There are newer tools that provide graphical representation of this data as well (e.g. - show the tree as sort of a topographical map, with hotspots as peaks, and the call tree as geological strata)
Of course, if your program has a giganto routine that it never leaves, you might not learn much -- "do_all_the_work_inline() is executing 95% of the time!", unless you start looking at things at the line number level. Blech.
For an (SQL) database, you would want to look at something called a query planner (e.g. - "explain" command).
It describes the order that tables are accessed, which indices (if any) are used for each access, "and so on".
It describes the order that tables are accessed, which indices (if any) are used for each access, "and so on".
Profiling data tells you how much time your software spends running each part of the program.
At the highest level, you might just know "we spent 25% of time reading from the disk, 10% writing to network", etc.
At the lowest level, you can know exactly how many times each instruction was executed, and how long it took, and the call stack of each execution.
Normally somewhere in between. The appropriate level depends on what you're trying to optimise.
At the highest level, you might just know "we spent 25% of time reading from the disk, 10% writing to network", etc.
At the lowest level, you can know exactly how many times each instruction was executed, and how long it took, and the call stack of each execution.
Normally somewhere in between. The appropriate level depends on what you're trying to optimise.
I don't agree that abstraction layers inherently hide inefficiencies.
In C++, `unique_ptr` is an "abstraction layer" with no runtime overhead, that will probably make your code faster due to the better safety.
In C++, `unique_ptr` is an "abstraction layer" with no runtime overhead, that will probably make your code faster due to the better safety.
I don't see such claim anywhere in the article (your "inherently" vs. author's "often").
That's true, but he makes the point to remove the abstractions, and I'd argue that no, fix the abstractions.
Actually, that's not the point I was trying to make!
I was trying to say: "consider the abstractions, and make them more leaky when necessary" (e.g. allow a flag to say "don't do the expensive calculation"), not "remove the abstractions".
I was trying to say: "consider the abstractions, and make them more leaky when necessary" (e.g. allow a flag to say "don't do the expensive calculation"), not "remove the abstractions".
Depends on what kind of abstraction layer you're talking about.
The traditional BLL/BOL/DAL architecture is probably the worst. It aims for some kind of fantasy "separation of concerns" between the business layer and the data access layer, but many concerns simply can't be classified as exclusively business concerns or exclusively data access concerns. When people attempt to introduce and enforce such a separation, they end up with all sorts of performance problems.
The traditional BLL/BOL/DAL architecture is probably the worst. It aims for some kind of fantasy "separation of concerns" between the business layer and the data access layer, but many concerns simply can't be classified as exclusively business concerns or exclusively data access concerns. When people attempt to introduce and enforce such a separation, they end up with all sorts of performance problems.
They don't necessarily hide inefficiencies, but they do give them a place to hide.
It can be looked at on a few levels.
One is the "simple things" that waste a few seconds here and a few seconds there and pretty soon we are talking about hours.
Another is the algorithm that does more work than it has to do, scales poorly, etc.
In my mind "encapsulation" and "abstraction" do tend to be leaky when it comes to things like: performance, security, thread safety, scalability, etc.
It can be looked at on a few levels.
One is the "simple things" that waste a few seconds here and a few seconds there and pretty soon we are talking about hours.
Another is the algorithm that does more work than it has to do, scales poorly, etc.
In my mind "encapsulation" and "abstraction" do tend to be leaky when it comes to things like: performance, security, thread safety, scalability, etc.
Or you can make the logging part asynchronous (provided the server itself is not maxed out) and cut that 30% of request time.