Coding Horror: Hardware is Cheap, Programmers are Expensive(codinghorror.com)
codinghorror.com
Coding Horror: Hardware is Cheap, Programmers are Expensive
http://www.codinghorror.com/blog/archives/001198.html
59 comments
I applied to a job in the spring of 2006 and when they were showing me around the development area I noticed that everyone had single smallish CRTs and what seemed like older computers. I asked about it and they told me that most people had P3-500's! Unsurprisingly their salary offer was low also and I did not take the job.
I took that job and for 5 years each day when I arrived at work I would have to go around looking for somebody that had carelessly wandered off to the bathroom and steal their chair. Eventually I just settled on one that was missing two wheels and an arm rest since it tended to not mysteriously vanish as often while I was at home sleeping or away eating lunch.
Yeah, at that point you can pretty much conclude that the company is doing something incredibly wrong, and it's only a matter of time before it bites them.
If you have crappy pay and perks, then you won't attract good talent, and whatever good talent you had will eventually leave. Without good talent, it is highly unlikely you'll have a good app. If your app sucks, you will slowly bleed off customers, and revenue.
Of course, that's not a scientific fact. For exceptions start with: Contracts, Government.
If you have crappy pay and perks, then you won't attract good talent, and whatever good talent you had will eventually leave. Without good talent, it is highly unlikely you'll have a good app. If your app sucks, you will slowly bleed off customers, and revenue.
Of course, that's not a scientific fact. For exceptions start with: Contracts, Government.
I quit a job a couple of years ago where (I kid you not) I had no desk, no allocated seating, and no PC. I was expected to bring in my own laptop. Whenever there was a director's meeting, we all had to surrender our communal office and hide in the server room. Duration of job: less than six months. It would have been shorter if I'd know what assholes they were (I had some idea they might sort out the problems if I waited a while). Worst. Company. Ever.
I worked for a state agency a long, long time ago. I was doing Perl-CGI web app, but I was only given a tiny table, a wooden chair, and a little teeny green-screen CRT I could use to rlogin and have shell access. This was actually a machination by the current system administrator who wanted the project to fail because we were impinging on his territory. (He wanted all of that State's Public TV related online publishing to be on an old-style text BBS he had set up.)
Well, my project was wildly successful anyways and I ended up being moved to a proper office and he ended up getting a different job.
Well, my project was wildly successful anyways and I ended up being moved to a proper office and he ended up getting a different job.
...and you're not bothering to tell us who it is, so someone from HN might get suckered into working for them next month.
Well, I'm guessing most of you guys are in the US anyway. But if you find yourself in the county of Somerset, UK, and get a chance at an interview with a company providing IT services to a well-known defence contractor, and they tell you it's web development, systems administration and "a bit" of support work, just ask them to define "a bit" rather more precisely.
I can't be too specific because we have some rather repressive libel laws over here.
I can't be too specific because we have some rather repressive libel laws over here.
Awesome :) Me, I'm in Argentina.
This is a ridiculous myth that I wish we could just put to rest once and for all. If your overpaid programmers are writing O(n^2) algorithms then that cheap hardware is going to get expensive real quick. Even if hardware was free you still need to put it somewhere, power it, cool it and so on. There's no excuse for sloppy coding.
If your overpaid programmers are writing O(n^2) algorithms then that cheap hardware is going to get expensive real quick.
Well, that depends on the value of n, right? Not every piece of software is Google or Twitter: They don't all have a userbase that grows virally to encompass every breathing human in existence, plus animals and robots. Especially if you charge the users money.
Well, that depends on the value of n, right? Not every piece of software is Google or Twitter: They don't all have a userbase that grows virally to encompass every breathing human in existence, plus animals and robots. Especially if you charge the users money.
Not every piece of software is Google or Twitter:
Worse, almost all software isn't like a Google or Twitter. In almost all cases hardware is vastly cheaper than programmers. The Googles and Twitters of the world are extreme outliers and are no where near the median.
Worse, almost all software isn't like a Google or Twitter. In almost all cases hardware is vastly cheaper than programmers. The Googles and Twitters of the world are extreme outliers and are no where near the median.
I don't believe that's true. Google sure, but twitter is just another message routing system, like any one of the IM networks.
The average irc server probably handles more than twitter - certainly email software does.
There's a ton of software that needs to scale, and be able to handle a large volume of data.
The average irc server probably handles more than twitter - certainly email software does.
There's a ton of software that needs to scale, and be able to handle a large volume of data.
There's a ton of software that needs to scale
Yes, but there's vastly more that doesn't, you need to stop thinking that most software runs on the Internet and has tons of users, it doesn't. The vast majority of software is written for businesses and runs either on an intranet or is available on the internet to the users of that business.
For every public app you know about that needs to scale there are a hundred you don't know about that don't need to scale. Most programmers work on systems that will never need to scale, EVER. It is a tiny tiny minority of programmers that actually work on systems that become publicly popular on the Internet and actually face scaling issues.
Secondly, we're talking about programmers here, not the software, most email systems use existing mail software written by that tiny tiny minority of programmers who write such systems. The vast majority of programmers will NEVER write a mail system, a chat system, or anything that handles large volumes of data or users. Most of them of doing one thing, biz apps for in house projects at big and small companies exposing relational data to in house users.
How many IM networks are there? Now how many programmers are there? I'll say it again, most software does not need to scale. Because a lot of software does need to scale does not make that statement in any way false because a shit load more doesn't need to scale.
Yes, but there's vastly more that doesn't, you need to stop thinking that most software runs on the Internet and has tons of users, it doesn't. The vast majority of software is written for businesses and runs either on an intranet or is available on the internet to the users of that business.
For every public app you know about that needs to scale there are a hundred you don't know about that don't need to scale. Most programmers work on systems that will never need to scale, EVER. It is a tiny tiny minority of programmers that actually work on systems that become publicly popular on the Internet and actually face scaling issues.
Secondly, we're talking about programmers here, not the software, most email systems use existing mail software written by that tiny tiny minority of programmers who write such systems. The vast majority of programmers will NEVER write a mail system, a chat system, or anything that handles large volumes of data or users. Most of them of doing one thing, biz apps for in house projects at big and small companies exposing relational data to in house users.
How many IM networks are there? Now how many programmers are there? I'll say it again, most software does not need to scale. Because a lot of software does need to scale does not make that statement in any way false because a shit load more doesn't need to scale.
The vast majority of software is written for businesses and runs either on an intranet or is available on the internet to the users of that business.
That doesn't mean a thing in terms of scalability. One user can max out a fast machine with OLAP cubes or other DSS. Or any sort of simulation or modeling. Think in terms of datasets, not in terms of users (all many users are is a large dataset with a requirement for a quick turnaround of each computation).
That doesn't mean a thing in terms of scalability. One user can max out a fast machine with OLAP cubes or other DSS. Or any sort of simulation or modeling. Think in terms of datasets, not in terms of users (all many users are is a large dataset with a requirement for a quick turnaround of each computation).
If an app only has one user, it doesn't need to scale, it needs to perform. Scalability is all about users and only loosely related to performance. Datasets are irrelevant.
I don't know what dictionary you're using, but here are a few definitions from mine:
scalable - How well a hardware or software system can adapt to increased demands.
scalable - the ability of a product or network to accommodate growth
The ability to scale hardware and software to support larger or smaller volumes of data and more or less users.
the ability of a computer system to shrink or grow as requirements change
scalable - How well a hardware or software system can adapt to increased demands.
scalable - the ability of a product or network to accommodate growth
The ability to scale hardware and software to support larger or smaller volumes of data and more or less users.
the ability of a computer system to shrink or grow as requirements change
That's not true. It needs to scale from 1M of data to 1G to 1T and beyond. You seem to think that scalability only refers to concurrency. It doesn't. It means that you're as close as possible to O(n) or better yet O(1) as your workload increases, whatever that workload is.
Twitter is not "just a messaging system". See:
http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi...
(Many thanks to Eran Hammer-Lahav for writing this so that I don't have to.)
http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi...
(Many thanks to Eran Hammer-Lahav for writing this so that I don't have to.)
I respectfully disagree. It's a simple message passing system, of which there are a ton of examples. This isn't rocket science.
The problem is that where once it would have been written in c as a server, it's now written with big databases and 'popular' languages - so it's harder to scale.
The problem is that where once it would have been written in c as a server, it's now written with big databases and 'popular' languages - so it's harder to scale.
It would have taken longer to get working in C, which would have dramatically increased the probability that the project would have gotten killed before it saw users (it sounded like a pretty stupid site to me at first!), and it would be harder to rev the UI and features as often as they did before they hit the scaling problems.
But a message server in Ruby with Eventmachine wouldn't have been that hard, would it?
But a message server in Ruby with Eventmachine wouldn't have been that hard, would it?
True. IBM have basically solved messaging with MQseries running on OS/390. All that remains is do that, only cheaper.
Proprietary message queues are a huge problem (and a huge business).
There are better ways: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protoc...
For a good introduction check out http://google-ukdev.blogspot.com/2008/09/rabbitmq-tech-talk-...
There are better ways: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protoc...
For a good introduction check out http://google-ukdev.blogspot.com/2008/09/rabbitmq-tech-talk-...
If you can determine an upper bound on n, then your algorithm is technically O(1), and you don't have to worry about scaling.
If you want to be able to handle any value of n, and expect it to be large, you should care about O(n^2) algorithms. You don't have to necessarily get rid of them all at once, but you should at least know where they exist and have them listed as possible pain points or places to start optimizing.
If you want to be able to handle any value of n, and expect it to be large, you should care about O(n^2) algorithms. You don't have to necessarily get rid of them all at once, but you should at least know where they exist and have them listed as possible pain points or places to start optimizing.
It's not just about users because there are no users; there's just data. Any data, the same thing applies.
Also, I'd add that designing an app to scale out effectively is one of the harder challenges that most developers face in the world of web apps. He seems to imply that you just by another few boxes and magic happens, which is not the case from what I've seen.
To be fair, the article is a little more balanced than the headline.
To be fair, the article is a little more balanced than the headline.
O(N^2) forgets that in most cases it's really more like K * O(N^2) + I and reducing the size of K can have dramatic impact because it tends to be more than 10,000 cycles. For a wide range of inputs a low latency N^2 can be faster than a high latency n * Log n because thrashing you cache is really bad.
The basic question you need to ask is how much are you spending on hardware / bandwidth vs coders and which is better to focus on.
The basic question you need to ask is how much are you spending on hardware / bandwidth vs coders and which is better to focus on.
That's really well put.
In this case, I'd say you need programmers to reduce the O(N^2), and hardware to reduce the K.
If you can reduce K by 50% for $5,000, then it makes almost no sense to use a programmer to reduce K. However, hardware will almost never save you from a programmer who gives you an O(N^2) instead of a log N solution.
This happened at my last job (actually, worse). Management decided I would be an "Architect" and train an offshore team to write the code (hmm... seems kind of fishy). We were building a system that would hold an equation with millions of constraints and variables, and we needed to be able to look up a variable value by name.
The code from the offshore team, no kidding, stored them in arrays. Unreal. I quit, for all kinds of reasons.
(yes, yes, many offshore teams could have written this code very well - the problem was not offshore programmers per se, it was that mgmt wanted top talent for cheap and was outsourcing its core product, and you don't get something for nothing).
In this case, I'd say you need programmers to reduce the O(N^2), and hardware to reduce the K.
If you can reduce K by 50% for $5,000, then it makes almost no sense to use a programmer to reduce K. However, hardware will almost never save you from a programmer who gives you an O(N^2) instead of a log N solution.
This happened at my last job (actually, worse). Management decided I would be an "Architect" and train an offshore team to write the code (hmm... seems kind of fishy). We were building a system that would hold an equation with millions of constraints and variables, and we needed to be able to look up a variable value by name.
The code from the offshore team, no kidding, stored them in arrays. Unreal. I quit, for all kinds of reasons.
(yes, yes, many offshore teams could have written this code very well - the problem was not offshore programmers per se, it was that mgmt wanted top talent for cheap and was outsourcing its core product, and you don't get something for nothing).
If you are going to pay big bucks to a programmer, you want one that does a little algorithm analysis before s/he puts fingers to keyboard, same with memory usage. I saw a major project go down the tubes because the architect thought that it scaled limitlessly but really it scaled n^2. Ain't good for a transaction protocol.
If you need to scale, you need to have a good architectural path from the beginning. I've talked to too many companies whose growth is constrained by the need to re implement from the ground up, right in the middle of the hockey stick growth. What I mean is a little bit of thought at the beginning, not months of agonizing or 9 tons of frameworks. Given that, you can throw hardware at it and let your programmers add value.
If you need to scale, you need to have a good architectural path from the beginning. I've talked to too many companies whose growth is constrained by the need to re implement from the ground up, right in the middle of the hockey stick growth. What I mean is a little bit of thought at the beginning, not months of agonizing or 9 tons of frameworks. Given that, you can throw hardware at it and let your programmers add value.
Yep, good programmers might be expensive, but in the long-term so is the energy needed to power overworked hardware from sloppy coding.
Although, when it comes to custom intranet apps, N is usually tiny.
The IT world is a lot bigger than websites. Monte Carlo simulations for example are constrained literally by power and cooling and the floor loading of your datacentre. If you aren't squeezing every last drop of compute out, you might as well not bother. Algorithms are everything.
[deleted]
When I started as a programmer in the late 70s the relationship between hardware costs and programmer costs were just about the reverse to those described by Jeff. It was always cheaper to optimise the code than it was to invest in new hardware. I must admit that it is hard to shrug off the frugal approach to the hardware that I adopted at the start of my developer career - I think it still pays off to err on the side of low hardware overhead whenever the option arises.
Hrm. I guess the guy cited in my online journal was really thinking ahead:
http://journal.dedasys.com/2008/12/04/the-economics-of-progr...
It's an extract from a paper from the 1950'ies by Backus (yeah, that Backus) saying pretty much exactly what Atwood says.
http://journal.dedasys.com/2008/12/04/the-economics-of-progr...
It's an extract from a paper from the 1950'ies by Backus (yeah, that Backus) saying pretty much exactly what Atwood says.
This argument is applicable only to code that runs on only a small number of machines. Using roughly his numbers, you can afford to pay somebody US$100 000 to work for a whole year to make your code 10% faster if that code uses 100% of a million dollars' worth of hardware, which is only 200 high-end US$5000 servers or gaming machines, or 2000 low-end servers.
But that's assuming that the benefit of the speed improvement is merely improved throughput resulting in data-center savings. If the issue is that your non-AJAX UI is frustratingly and inconsistently slow and so your ex-customers have switched to your competitors' AJAX app because it has consistent 200ms response times when they click on buttons, it may not matter if their app actually uses more CPU cycles than yours (because half of it is written in JavaScript and your ex-customers aren't running Chrome yet.) If your game gets 20 frames per second and your competitor's game gets 35 frames per second with nicer-looking graphics, you're going to lose sales, even if the customers run the game for only 10 hours each.
And that's why gamer games and desktop apps are still written largely in C, C++, or Java with some Lua, and Google's large-scale apps are mostly written in C++ and Java, while small-scale server-based applications and the stuff developed by the in-house IT guys are written in all kinds of higher-productivity, lower-performance languages.
But that's assuming that the benefit of the speed improvement is merely improved throughput resulting in data-center savings. If the issue is that your non-AJAX UI is frustratingly and inconsistently slow and so your ex-customers have switched to your competitors' AJAX app because it has consistent 200ms response times when they click on buttons, it may not matter if their app actually uses more CPU cycles than yours (because half of it is written in JavaScript and your ex-customers aren't running Chrome yet.) If your game gets 20 frames per second and your competitor's game gets 35 frames per second with nicer-looking graphics, you're going to lose sales, even if the customers run the game for only 10 hours each.
And that's why gamer games and desktop apps are still written largely in C, C++, or Java with some Lua, and Google's large-scale apps are mostly written in C++ and Java, while small-scale server-based applications and the stuff developed by the in-house IT guys are written in all kinds of higher-productivity, lower-performance languages.
He is really writing this from the perspective of web applications. If you app is humming along nicely on a single EC2 instance, it isn't even worth an HOUR of your time to DOUBLE the capacity of your app.
The last big app I was in charge of ran 1M pageviews a day on $500/month. At my standard rate, it isn't even worth a day of my time to double the capacity of my app on the same hardware.
He certainly isn't saying don't optimize. He's simply saying that scaling web apps STARTS with throwing hardware at the problem. Where and when you cut over to developer optimization depends on your -actual- costs, size, and growth.
The last big app I was in charge of ran 1M pageviews a day on $500/month. At my standard rate, it isn't even worth a day of my time to double the capacity of my app on the same hardware.
He certainly isn't saying don't optimize. He's simply saying that scaling web apps STARTS with throwing hardware at the problem. Where and when you cut over to developer optimization depends on your -actual- costs, size, and growth.
Yes, I agree, although it might be worth an hour of your time to improve the app's responsiveness if it makes the customers happier.
What this article lacks is an eye toward things that must be optimized when you get to a certain level.
For example, when you replicate databases, the replication overhead grows quadratically (n^2) to the number of servers. So, eventually you can't just throw hardware at it since eventually it's all overhead replicating between the servers and you can do any selects/updates/deletes. Most people will never get to this level and the article is probably written for people that won't have to deal with this.
So, at a certain level, you need to optimize your code to shard data, avoid joins, etc. because you simply can't throw hardware at it.
Most people will never get to that level so throwing hardware at the problem is often the answer, but if you have a lot of quadratic algorithms in your code, you're going to hit scaling problems that will make throwing money at it expensive to the point where you'll loose money.
For example, when you replicate databases, the replication overhead grows quadratically (n^2) to the number of servers. So, eventually you can't just throw hardware at it since eventually it's all overhead replicating between the servers and you can do any selects/updates/deletes. Most people will never get to this level and the article is probably written for people that won't have to deal with this.
So, at a certain level, you need to optimize your code to shard data, avoid joins, etc. because you simply can't throw hardware at it.
Most people will never get to that level so throwing hardware at the problem is often the answer, but if you have a lot of quadratic algorithms in your code, you're going to hit scaling problems that will make throwing money at it expensive to the point where you'll loose money.
Do software programmers really worry too much about optimization? If that's the case, why are so many programs slow as a crippled sloth? Why does it take 10 seconds to open a freaking email client, when 1980s technology can do this in less time than is humanly noticeable?
Well, no, they don't bother, because people keep telling them that hardware is cheap and they're too expensive to work on anything that isn't new features.
That's got more to do with the size of all the frameworks & libraries loaded in, than the actual code of the app.
A modern desktop app has a lot more expected of it than one from the 1980s (GUI, copy-paste with styled text & embedded objects, drag & drop, undo, etc), and that code costs time to link in, use, and let run.
A modern desktop app has a lot more expected of it than one from the 1980s (GUI, copy-paste with styled text & embedded objects, drag & drop, undo, etc), and that code costs time to link in, use, and let run.
This argument works only for enterprise/server development, where hardware is under your control.
When developing for the consumer market, hardware is not "expensive", hardware is simply not available: users control their hardware, and even the argument of "ever increasing gigabytes and gigahertz" doesn't work anymore, as more and more prefer to trade speed for increased portability or (surprise!) smaller price.
When developing for the consumer market, hardware is not "expensive", hardware is simply not available: users control their hardware, and even the argument of "ever increasing gigabytes and gigahertz" doesn't work anymore, as more and more prefer to trade speed for increased portability or (surprise!) smaller price.
There's an issue of need satiation. Once a need is met, further improvements can't be absorbed. Like water when you aren't thirsty, it doesn't matter.
Hardware helps performance - but this only matters when more performance is needed. For example, if features or usability are needed, extra performance doesn't help. This is obvious, but it's surprisingly easy to get caught up in improvements that are better, but that don't matter.
For productivity of programmers, faster hardware only helps when performance is a problem. The article says: If [it] makes them merely 5% more productive... - that's an "if". Will a faster PC make you more productive? These days, compilation is too quick for me to squeeze in a swordfight. For other hardware (eg large screens) there is also a limit beyond which extra real estate isn't useful (Joel on Software talked about several of his coders that don't use their second monitor). It depends on the task, of course.
Faster is always cool and impressive. But like a 300 miles per hour station wagon, will it help you?
Hardware helps performance - but this only matters when more performance is needed. For example, if features or usability are needed, extra performance doesn't help. This is obvious, but it's surprisingly easy to get caught up in improvements that are better, but that don't matter.
For productivity of programmers, faster hardware only helps when performance is a problem. The article says: If [it] makes them merely 5% more productive... - that's an "if". Will a faster PC make you more productive? These days, compilation is too quick for me to squeeze in a swordfight. For other hardware (eg large screens) there is also a limit beyond which extra real estate isn't useful (Joel on Software talked about several of his coders that don't use their second monitor). It depends on the task, of course.
Faster is always cool and impressive. But like a 300 miles per hour station wagon, will it help you?
It's rare that I come across a problem where resource and speed constraints do not become an issue. Javascript and the DOM and latency has been an issue with client side web-apps. DB bottlenecks have been an issue on the serverside. Numerical performance and AI performance is an issue in CAD/CAM/CAE games and scientific analysis. In every scenario there have been algorithmic and software level improvements that have speed up the system by a factor of ten or more. Certainly it plateaus after a while, but one cannot hope to throw hardware at complicated problems and expect them to be solved without some serious elbow grease on the part of the programmer. Programs become increasingly more ambitious, and expand to fill the resources allocated to them: a kind of Hacker's Parkinson's law.
Simplest method is still the best:
Write something that works. Run it at production loads. If it's not good enough, work on it. Don't worry about speed until you hit that. Only exception is if you know N's going to be big enough to be problematic.
In that case, do just enough optimization to make it reasonable.
Why? Optimization and maintainability* are often at opposing ends. It's best to avoid the costs of it when it's not needed, and the increasing capacities of computers should make you bias towards maintainability.
* and ease of debugging, and concerns for low software complexity, and likelihood that it will be well documented, and .....
Write something that works. Run it at production loads. If it's not good enough, work on it. Don't worry about speed until you hit that. Only exception is if you know N's going to be big enough to be problematic.
In that case, do just enough optimization to make it reasonable.
Why? Optimization and maintainability* are often at opposing ends. It's best to avoid the costs of it when it's not needed, and the increasing capacities of computers should make you bias towards maintainability.
* and ease of debugging, and concerns for low software complexity, and likelihood that it will be well documented, and .....
Am I the only one thinking those salary figures are around 30-40K too low?
It's the whole US, it isn't broken down by geographic region or anything.
But even then, 30-40K is weak. 40K should be your starting salary. Anything less, and you're being taken.
I practise the 'path' method of speed optimisation: If you can not execute a piece of heavy code every 10 in 100 runs, you save a lot, just by a simple data/value check before it.
As for memory opts, pass by reference instead of value (in C++).
I've sped up code by a factor of hundreds by doing some crazy stuff (without going to assembly).
As for memory opts, pass by reference instead of value (in C++).
I've sped up code by a factor of hundreds by doing some crazy stuff (without going to assembly).
Yep, memoization is often a big win, and so easy to do in Python with a decorator.
Managing servers can be a real pain in the ass. If you can get everything to work on one server, life is much simpler.
i'd say programming environment (unix/mac, vim/emacs) and skill levels (algo, data structure, etc) matter more than hardware
it costs less to hire a skillful (but expensive) developer on ok machine than less skillful one(s) on expensive machine(s)
the 10-100x gain myth on skill is pretty much true
it costs less to hire a skillful (but expensive) developer on ok machine than less skillful one(s) on expensive machine(s)
the 10-100x gain myth on skill is pretty much true
[deleted]
This is probably the most overlooked reason to get at least semi-good hardware. Part of the reason I quit my last job was because of management's stubborn refusal to get us tools that would make us be more productive.