Ask HN: Growing faster and starting to have scaling issues. Where to go from here?
9 comments
First, congratulations! Having too many customers is a great problem to have.
Now, you need to unpick the legacy code that makes up your current MVP and turn it into a solid, scalable system. Recognise that this is not going to happen overnight. You should also be wary of trying to fix it with a single "big bang" solution (and be wary of any prospective CTO that might suggest that).
The good news is that all technology companies go through this when they reach a certain scale, so techniques for re-engineering legacy systems are well-known, as are techniques for managing that re-engineering. You can Google these things, talk to experienced engineers, etc. - lots of people have war-stories.
Here's my advice:
1. Make re-engineering a process. Never stop. Welcome change. No code is sacred.
2. Split off a small piece of functionality from your app, such as formatting email or printing invoices or whatever. Write a small web-service to do just that functionality, nothing more.
3. Set up automated testing for your new web-service.
4. Set up monitoring of your new web-service.
5. Run your new web-service in parallel with your legacy system. Be able to switch back to legacy if your monitoring shows any issues.
6. Evaluate how well you executed steps 1-5. Feed changes back into your process. - Goto 1
You can use this model with outsourced development, and switch devs after each iteration if you feel that is necessary. After a few iterations, you should be able to use several devs to build web-services in parallel.
Good luck.
Edit: formatting
Now, you need to unpick the legacy code that makes up your current MVP and turn it into a solid, scalable system. Recognise that this is not going to happen overnight. You should also be wary of trying to fix it with a single "big bang" solution (and be wary of any prospective CTO that might suggest that).
The good news is that all technology companies go through this when they reach a certain scale, so techniques for re-engineering legacy systems are well-known, as are techniques for managing that re-engineering. You can Google these things, talk to experienced engineers, etc. - lots of people have war-stories.
Here's my advice:
1. Make re-engineering a process. Never stop. Welcome change. No code is sacred.
2. Split off a small piece of functionality from your app, such as formatting email or printing invoices or whatever. Write a small web-service to do just that functionality, nothing more.
3. Set up automated testing for your new web-service.
4. Set up monitoring of your new web-service.
5. Run your new web-service in parallel with your legacy system. Be able to switch back to legacy if your monitoring shows any issues.
6. Evaluate how well you executed steps 1-5. Feed changes back into your process. - Goto 1
You can use this model with outsourced development, and switch devs after each iteration if you feel that is necessary. After a few iterations, you should be able to use several devs to build web-services in parallel.
Good luck.
Edit: formatting
Assuming you are on AWS you could look into using some of their tools like Opsworks to help deal with the growing pains.
Its a temporary solution since you would just be using more servers to alleviate the root cause rather than optimizing the code. Opsworks will let you deploy additional servers easily and a contractor can do all the technical setup for you. You can queue servers based on load or time of day and it can run all of your php/database servers for you.
Hope it helps and you can use something like New Relic to help notify you of downtime or performance gaps in your application.
Its a temporary solution since you would just be using more servers to alleviate the root cause rather than optimizing the code. Opsworks will let you deploy additional servers easily and a contractor can do all the technical setup for you. You can queue servers based on load or time of day and it can run all of your php/database servers for you.
Hope it helps and you can use something like New Relic to help notify you of downtime or performance gaps in your application.
Throw money at the problem, ie scale up your servers. Move to a VPS/Cloud if you're on your own hardware. The Database is often the hardest part to scale, use services such as Amazon RDS or Heroku Postgres.
Add monitoring to wake you up if the site is down (pingdom is great).
Add monitoring to wake you up if the site is down (pingdom is great).
Umm.. the reverse advice would be more helpful. Move to your own hardware if you're on VPS/cloud.
Do not take external money. The longer you can bootstrap the better. I know you've heard this a bajillion times, and I know the allure of a few mil in the bank. It's not worth it. Hold off as long as you possibly can.
We've used Google's AppEngine to scale. It's not the easiest env to work with, but at 30+ staff and millions in revenue we have yet to hire a sysadmin (ie. overhead!). When load increases we just get charged for more resources.
Sounds like you have to parallelize: start MVP rebuild, and do mitigation on current platform. Four years in we are on the fifth major rev of our core processing system; point being that things change and you need to prepare for ongoing evolution.
[deleted]
Hiring a contractor is a good idea, but make sure they are hired with the potential to become a technical cofounder.
My suggestion would be to hire someone who can rewrite the PHP code into something more maintainable like Javascript, Ruby, or Python. And make sure they have a plan for scalable infrastructure (either via a PaaS like Heroku/App Engine or if they have the chops, a real DevOps infrastructure).
My suggestion would be to hire someone who can rewrite the PHP code into something more maintainable like Javascript, Ruby, or Python. And make sure they have a plan for scalable infrastructure (either via a PaaS like Heroku/App Engine or if they have the chops, a real DevOps infrastructure).
Suggesting a code rewrite w/o any idea of what the code looks like or what the issues are is pretty awful.
PHP is a good solution for plenty of projects out there. There are plenty of high revenue operations that run on PHP. Like any other language it just needs to be well coded and maintained.
What's needed here is looking at the current state of the system, detecting obvious issues and determining the course of action based on that + expected growth.
PHP is a good solution for plenty of projects out there. There are plenty of high revenue operations that run on PHP. Like any other language it just needs to be well coded and maintained.
What's needed here is looking at the current state of the system, detecting obvious issues and determining the course of action based on that + expected growth.
What possible advantage would "real" devops infrastructure provide? Unless you're doing something so far out if the ordinary (or infrastructure is your advantage) there is zero value in building your own early on.
You're suggesting they find a CTO/architect/php developer, all in one... That's not reality.
In a big comanies all three roles need a very different skill set, because of politics, splitting saying and doing and so on. In a start up environments skills needed for all three roles are very similar.
might look into this language. Facebook used it to scale. http://hacklang.org/
might look into this language. Facebook used it to scale. it works with PHP. http://hacklang.org/
[deleted]
It's depends on your scale problem. But I guess you mean perfromance (and not maintability or other) problem. There are a lot technics on different levels to save you for some time depending on your bottlenecks. Without concrete details of your stack nobody can give you "the answer". But in (my) practice, small improvements on right places can speed up your site up to 80%. Depending on your client presentation, you can hide some latency by minimizing and merging css and js-stuff and put them to cdn and cache forever. Decide to master-slave your database, cache query results and so on. These relatively small changes give you some time to develop more deeply changes.
Note: using throwaway account for privacy reasons.
We're a two-person team running a bootstrapped startup - a community marketplace for a popular consumer good. We are currently experiencing faster growth and expect it to continue to increase a lot more (we’ve found a good way to grow). We now have over 80k members and nearing $1 million dollars worth of goods exchanged. The issue now is keeping up with the growth.
The site is written in PHP and the code isn’t exactly pretty – but we know how to grow and have a ton of potential for expanding to other products. We’ve had to hold back on some marketing because we don’t want to risk breaking things.
So far we’ve been outsourcing the development because we: a) Couldn’t afford a full time developer in the past and b) Didn’t know any developers that would fit as a co-founder where we’re from
We’ve now just moved to San Francisco to start networking and find a CTO/technical co-founder but it’s going to take a little time to find the right person.
We’re also looking at just hiring a contractor short term. We wanted to give at least a week or two of being here in case we can find someone networking first but recent growth has sped things up. This can definitely still be done but two things make it slightly harder: a) Although we’re making a good profit now we can’t really stay competitive with salary compared to other funded startups. b) We both have limited knowledge with programing but need to make sure we hire someone that has the right skill set and can fix the issues we're having.
Are we covering everything we could be doing? Does anyone have advice on where to go from here or been in a similar position?
Any advice is much appreciated! Or on the off chance you think you could help us out, please drop us a line!
[email protected]