Well, it depends on the application. Most people think about scaling far too early, and the same applies to configuration management. I am a fan of shortcuts, early and often... worry about things when you have things to worry about.
That being said, making some easy architecture choices early on can have an enormous effect on your sanity. It is probably worth it to bring someone in to hint you in the right direction after you have a prototype to show and have already started to make choices. This is basically where I helped Adrian.
If your app is struggling under success, you are ideally in a better position financially than most startups are at the beginning. This is when you bring someone in for longer term engagements, or potentially when you hire someone to work on this full time.
I think you are hitting on something that is really a need in the market right now. When you don't need a consultant, and you need 30-90 minutes of "how do I do this?" Q&A, it is hard to figure out who to talk to.
From my perspective, it is hard to find those engagements. I do a sad majority of my time in the sales side of things. Recently, I spoke to some folks at 10xmanagement and Adrian pointed me to anyfu, both of which are approaching this problem but in slightly different manners.
Lastly, you don't want the soup to nuts guy. This is your app, and if you aren't deeply involved with what is going on with it, you are putting yourself at a huge disadvantage post consulting engagement.
Huge fan of salt stack, and I may push Adrian in that direction. The changes we made here were minimal in the way he deploys, and with so many architecture changes behind the scenes, we decided less is more on that front.
Really loved working on this with you, and I'm excited to see what you come up with now that you have a much better understanding of AWS. Congrats on the re-launch!
This is 100% correct. We also generated static snapshots of the origin and mirrored to S3. In the instance of something going awry with our primary origin servers, we could fail back to S3 as an origin.
EDIT: I'll add that our origin offload for WWW hovered around 98%.
The subtlety you are referring to in #2 is that we moved from puppet for deployments to puppet to configure template nodes. We baked AMIs from those template nodes and deployed those, drastically shortening our time from boot to serving traffic.
A) That's a bit nuanced and depends on the application you are referring to (there are ~200 represented here), but if you had the scripts and the repos and built from a vanilla AMI, a box would boot and configure within a few minutes. We shortened the boot times greatly by using puppet to configure a template node which we then used to create a generic versioned AMI per autoscale environment and deployed with Asgard. If you had the AMIs, an EBS backed instance would boot and serve traffic in anywhere between 15 and 30 seconds or so.
B) For costs, you can check the FEC filings... I cannot recall what they were month to month, but it will be accurately reflected quarterly there, so you can get a rough estimate. It was like a kabillion times more in October 2012 than it was in June 2011. The important part for us is that the costs scaled with our demand.
That part is a a little more nebulous. When deploying the entire full stack of all applications is automated, rolling a new environment is very simple. So, for instance, if I wanted to load test a specific end point in our API, I would just build a new instantiation of the API and test against that in a protected and isolated environment.
So in reality, it might be 2 staging environments and 12 testing. There was no real set number of environments.
You are 100% correct. I honestly think the problem is the speed at which we are throwing this together right now. This is an exercise for us in rapid implementation. The team that put this together is the same team that launched chicago2011.org, an app that took 72 hours from conception to development to implementation. I consider this a massive accomplishment, especially considering we were working with government. Coming off of that success, we were pumped to push out potentially a real startup under similar time pressure. You are now seeing some of the sloppiness that exists at the edges of a project like that, and I too shared your sentiment when usehipster went back into stealth/hype mode. Using launchrock certainly doesn't help that since it is works really well as a vague hype machine.
I promise you that I will put more detail up to the public within 48 hours, and I would love to personally speak with you to explain exactly what we're doing. You can email me at [email protected] if want, I would love to hear more of your feedback.
Yes, we could have. However time is of the essence here, and we took all of the recommendations you see directly from the Chicago Police Department. They recommended a direct cash incentive would be the quickest and easiest way to find both of these guys.
Sorry dudes, we were told by the police that we cannot release any information as to _why_ we want to track these guys down. They said that in situations in the past, people have been sued for defamation.
I really really really wish I could tell you what is going on. For now, you will have to rely on the internet rumormill:
That being said, making some easy architecture choices early on can have an enormous effect on your sanity. It is probably worth it to bring someone in to hint you in the right direction after you have a prototype to show and have already started to make choices. This is basically where I helped Adrian.
If your app is struggling under success, you are ideally in a better position financially than most startups are at the beginning. This is when you bring someone in for longer term engagements, or potentially when you hire someone to work on this full time.
I think you are hitting on something that is really a need in the market right now. When you don't need a consultant, and you need 30-90 minutes of "how do I do this?" Q&A, it is hard to figure out who to talk to.
From my perspective, it is hard to find those engagements. I do a sad majority of my time in the sales side of things. Recently, I spoke to some folks at 10xmanagement and Adrian pointed me to anyfu, both of which are approaching this problem but in slightly different manners.
Lastly, you don't want the soup to nuts guy. This is your app, and if you aren't deeply involved with what is going on with it, you are putting yourself at a huge disadvantage post consulting engagement.