Their execution was poor. Start by depleting SF and then look elsewhere? Short-sighted at best.
They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over.
Thanks EwanToo - that makes perfect sense. A combination of some security monitoring system that notifies you of the vulnerabilities along with someone to update your system is needed. But what if the updates have dependencies, for instance, incompatible Ruby gems or so. At that point, do you have to make the tradeoff of security risk vs time to update all gems/resolve incompatibility issues/deal with bugs in latest release?
If it only takes 1 mistake, would having a hot backup or failover be a best practice, so that if something does happen, you can immediately channel traffic to a live site?