Should I Be Worried About Scaling?(shouldibeworriedaboutscaling.info)
shouldibeworriedaboutscaling.info
Should I Be Worried About Scaling?
http://www.shouldibeworriedaboutscaling.info
10 comments
Er, not exactly "NO".
There are 2 kinds of scaling.
If you're worried about being able to handle a volume of 10,000 on the first day, then the answer is probably "NO".
OTOH, if you're concerned about designing something industrial strength from the inside out, then the answer is probably "YES".
YES, you might as well build reuseable components now if you know you'll need them. If you know what you're doing, that shouldn't take any longer that writing the same code multiple times.
YES, you should write processor intensive code at a low level if you know you'll only have to rewrite it later. Again, doing it right in the first place shouldn't take any longer.
YES, you should build a flexible enough architecture if your specs and design aren't yet frozen in stone (whose are?)
YES, you should probably be doing lots of things right now to avoid doing them twice. You must use your professional judgement about which ones.
A blanket NO bypasses that judgement. That's not being fair to yourself.
There are 2 kinds of scaling.
If you're worried about being able to handle a volume of 10,000 on the first day, then the answer is probably "NO".
OTOH, if you're concerned about designing something industrial strength from the inside out, then the answer is probably "YES".
YES, you might as well build reuseable components now if you know you'll need them. If you know what you're doing, that shouldn't take any longer that writing the same code multiple times.
YES, you should write processor intensive code at a low level if you know you'll only have to rewrite it later. Again, doing it right in the first place shouldn't take any longer.
YES, you should build a flexible enough architecture if your specs and design aren't yet frozen in stone (whose are?)
YES, you should probably be doing lots of things right now to avoid doing them twice. You must use your professional judgement about which ones.
A blanket NO bypasses that judgement. That's not being fair to yourself.
I agree that the answer is not exactly "No", but I disagree with most of your points.
Building reusable components, coding at a low level and building a flexible architecture all take considerably more effort than their alternatives, IMHO.
That being said, it makes sense to have the big picture in mind for the first time around and to take some small considerations on the way.
Avoid doing inherently bad things like schlepping around tons of unneeded data and not doing some simple caching.
Like the recent post about embracing technological debt, much of the early work may be discarded before it see the light of day so no point in over investing.
Rather that doing lots of things right the first time, I spend more effort not tightly coupling the pieces so that when you have to write something twice (and you will), it's a clean separation. I'd get the processor intensive stuff done with as little effort as possible (and likely barely fast enough), but in a way that I could rip that part out and plug in some native code when the time came.
Building reusable components, coding at a low level and building a flexible architecture all take considerably more effort than their alternatives, IMHO.
That being said, it makes sense to have the big picture in mind for the first time around and to take some small considerations on the way.
Avoid doing inherently bad things like schlepping around tons of unneeded data and not doing some simple caching.
Like the recent post about embracing technological debt, much of the early work may be discarded before it see the light of day so no point in over investing.
Rather that doing lots of things right the first time, I spend more effort not tightly coupling the pieces so that when you have to write something twice (and you will), it's a clean separation. I'd get the processor intensive stuff done with as little effort as possible (and likely barely fast enough), but in a way that I could rip that part out and plug in some native code when the time came.
My favorite examplei is naked SQL vs an ORM. The ORM will never be able to write queries that perform as well as if you write them all by hand. But, you'll be able to crank out functionality much faster.
Eventually, if you're lucky, you'll have enough traffic to warrant going back and rewriting some of the ORM calls using a SQL query optimized by hand.
Lastly, 99% of the time, the query you end up optimizing will not be the one you expected to when you started writing the app. That's because between when you started and now, the direction has changed significantly and you finally know what it is that you're really building.
Eventually, if you're lucky, you'll have enough traffic to warrant going back and rewriting some of the ORM calls using a SQL query optimized by hand.
Lastly, 99% of the time, the query you end up optimizing will not be the one you expected to when you started writing the app. That's because between when you started and now, the direction has changed significantly and you finally know what it is that you're really building.
Clearly you have good points.
But the URL wasn't ShouldIBeUsingProfessionalJudgmentAboutScaling.info, it was "worry"
http://en.wikipedia.org/wiki/Worry
But the URL wasn't ShouldIBeUsingProfessionalJudgmentAboutScaling.info, it was "worry"
http://en.wikipedia.org/wiki/Worry
semantics
If we're not arguing about semantics, then it's just syntax. Did you mean: nuance?
No, arguing about semantics does not mean what you think it does! It means arguing about what words mean instead of arguing substantively about the meanings themselves.
I unkilled this because it had a few interesting comments on it, but please stop submitting these gag sites that just have "yes" or "no" on the page.
So this is how we make arguments now? Reserve a complete-sentence domain name and put up a yes/no assertion?
Damn, I waited way too long for that page to not load...
Verizon just gave me one of their internet-breaking parking pages. Damn them.
you should create shouldibeworriedaboutscaling.com, SEO the crap out of it, and have it say yes
Quick answer: just don't prematurely screw it up. Some features and business ideas just don't scale.
The quickest way to check is to imagine you have everything in a big dumb (semi-normalized, but not fantastic) database. Eyeball the queries you're going to use. Then figure out how much work the database has to do to execute those queries. Also consider read/write rates and contention.
It's back-of-the-envelope, but it's pretty good.
The quickest way to check is to imagine you have everything in a big dumb (semi-normalized, but not fantastic) database. Eyeball the queries you're going to use. Then figure out how much work the database has to do to execute those queries. Also consider read/write rates and contention.
It's back-of-the-envelope, but it's pretty good.
One of the first concerns I have about a site is performance. Not scaling , but making sure the site itself operates very quickly. I have found that the path to getting a fast load time often leads to a reasonable level of scaling.
Doesn't it also depend on who 'I' am? If I am part of a tiny startup that is going to launch a twitter clone, then no, I guess not. But if I am at Facebook or Twitter or Google and I am adding a new feature to the app, then hell, yes. You will have millions of users on Day 1.
Getting irritated at work with talk about "we need to be ready, just in case"
Open the floodgates...and watch in despair as only a trickle comes through. Welcome to the real world.
Update: Wait until you get home to do something like this. Already got in trouble for it :( Need to remember: "Think then act"
Open the floodgates...and watch in despair as only a trickle comes through. Welcome to the real world.
Update: Wait until you get home to do something like this. Already got in trouble for it :( Need to remember: "Think then act"
That's very true. Plenty of 'start-ups' never make it past their 10,000th signup.
My approach is to throw stuff out there as soon as possible and see what gets traction and what does not. Plenty of time to concentrate on the few that do pan out that way.
Good ideas are two a penny, in the last 10 years out of all the websites we've built for ourselves and for customers the number of duds is staggering. A good idea in and off itself is not a guarantee that something will take off. Some of the most well thought out things I've built have totally flopped and some of the midnight hacks still give me bread on the table a decade later.
So my approach now is to keep the first implementation as thin as I think I can get away with without people noticing how solid something is.
Knowledge about building scalable systems is invaluable though when you do hit the jackpot.
My approach is to throw stuff out there as soon as possible and see what gets traction and what does not. Plenty of time to concentrate on the few that do pan out that way.
Good ideas are two a penny, in the last 10 years out of all the websites we've built for ourselves and for customers the number of duds is staggering. A good idea in and off itself is not a guarantee that something will take off. Some of the most well thought out things I've built have totally flopped and some of the midnight hacks still give me bread on the table a decade later.
So my approach now is to keep the first implementation as thin as I think I can get away with without people noticing how solid something is.
Knowledge about building scalable systems is invaluable though when you do hit the jackpot.
Very, very true. I've worked for more than a few companies that built multiply-redundant, highly scalable everything, and then ended up with ten small paying customers.
Don't tell me shouldibeworriedaboutscaling.com was taken?
They wanted too much $$ for it :)
It's not funny enough to warrant $10
.info cost me $1 w/o any coupons
It's not funny enough to warrant $10
.info cost me $1 w/o any coupons
I think all you have to do is add an alias in your apache configuration and it should work:
Serveralias shouldibeworriedaboutscaling.com
Serveralias www.shouldibeworriedaboutscaling.com
Let me know when you move your address so I can update the dns ;)
Serveralias shouldibeworriedaboutscaling.com
Serveralias www.shouldibeworriedaboutscaling.com
Let me know when you move your address so I can update the dns ;)
It's not funny enough to warrant $1. You got ripped off.
The first guy to set up one of these domains was pretty clever. Anyone else who had the idea before they saw one gets a free pass. But the rest of them are just lame. Kinda like milliondollarhomepage2.com et al.
The first 20 times I saw this, it was chuckleworthy. Now it's just old and spent.
haha OK, so it's still possible to make it funny.
[deleted]
[deleted]
We did this. Now our web servers sit at around 5% usage and we have to pay for the full rack that they sit in. We make plenty of cash however.
"Premature optimization is the root of all evil" is spot on.
"Premature optimization is the root of all evil" is spot on.
Aspiring "Next Facebook Founder" Non-Programmer: "You need to write the code so that it can scale to at least 3 million users in 6 months"
Contract Programmer Hired Off Craigslist: "Pardon me?"
ANFFNP: "If you look at the business plan, just after it defines your equity share in our little venture, you'll see that in week 20 we will have just over 1 million users. But by week 30 we will have 3 million users. After that, growth is exponential."
CPHOC: "Umm, I think we can worry about that later. I'm using Rails and Twitter was built on it. Sure they had some problems at first, but they managed to clear them up."
ANFFNP: "Well, I went to a startup pub night last week and this guy that wrote a URL shortener told me that Rails doesn't scale. I don't think we should use Rails."
CPHOC: "Okay... I guess I can use ASP.NET or PHP."
ANFFNP: "I heard at a FireFox 3.0 launch party last year not to use a Windows server because of security problems and that PHP powered Web 2.0. But our site's going to be Web 3.0, so PHP doesn't give us a competitive advantage."
CPHOC: "Well I've always wanted to learn Erlang..."
ANFFNP: "What's that?"
CPHOC: "It's this language that a phone company made to run its infrastructure. It apparently scales really well and was benchmarked at handling over 20 million processes once."
ANFFNP: "Excellent. I knew you were the right man for the job. I'll re-adjust our scalability numbers upward then. 20 million is about 6 times my original projections, so if I double them we should be good."
CPHOC: sigh...
Sorry, I'm in one of those moods today :-)