Farming hard drives: how Backblaze weathered the Thailand drive crisis(blog.backblaze.com)
blog.backblaze.com
Farming hard drives: how Backblaze weathered the Thailand drive crisis
http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/
11 comments
[deleted]
> this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use.
Important to remember that really it wasn't this one company, they were just in the rare situation of needing drives enough to try and buy lots, and not needing enough that they could go further up the food chain in the search of them.
Think of companies such as PC manufacturers, you'd be amazed what was going on behind the scenes.
Important to remember that really it wasn't this one company, they were just in the rare situation of needing drives enough to try and buy lots, and not needing enough that they could go further up the food chain in the search of them.
Think of companies such as PC manufacturers, you'd be amazed what was going on behind the scenes.
What would you have them do, just sit there and eventually be unable to provide their service to their customers?
i don't know. surprisingly enough, when it comes to difficult moral questions, i don't have all the answers.
Well, I'd suggest that the fact that they were financially able to carry out this operation suggests that they provide enough value to others for it to be a net gain. Money doesn't necessarily indicate value, of course, but it's a decent first approximation, especially for a business like theirs.
with that essential caveat, this is a very good point. My backup drive died in the midst of this crisis, and I specifically chose to use backblaze instead of buying a new drive. Obviously, it's not a 1-to-1 tradeoff in terms of functionality, but it was close enough for this data that I felt like I was better off paying them monthly than paying an inflated drive price upfront; they were drive farming for me by proxy.
[deleted]
I don't see an ethical issue here. BackBlaze came up with a creative solution to a supply chain problem. I see it like there not being enough food to go around; it doesn't make sense for everyone to starve to death. If BackBlaze had started the drive shortage, that'd be a different issue, but they found a way to survive in a tough spot.
I believe this is called "hustle".
It does not fit any definition of "hustle" that I am aware of, including these: http://en.wiktionary.org/wiki/hustle
It's definition 2a[1] on merriam-webster.com.
[1] http://www.merriam-webster.com/dictionary/hustle
a : to obtain by energetic activity <hustle up new customers>
[1] http://www.merriam-webster.com/dictionary/hustle
While I really enjoyed the story, I felt much the same way as you about the ethics. They justify their decision to run business as usual, supporting the same rate of natural growth, etc. by saying they didn't want to raise prices for their customers, but I don't think that's really a fair reason (eminent business failure would be a fair reason). There were other options:
1. freeze registration of new accounts and purchase only the drives necessary to keep existing contractual obligations
2. charge new accounts at a different rate, promising to drop the rate once drive prices returned to normal
3. change plan storage volume limits
The only reason they didn't pick any of these 3 options, as far as I can tell, is because all of them were bad for their profit margins and it would be nice if they were upfront about that as the real driver for their actions. It's not clear that their business would have failed had they chosen one of these options.
So they chose a path that would screw over the little guy needing a new hard drive in San Francisco. Perhaps I'm wrong, though, and the timing made it very difficult to act on these other options, or their cashflow couldn't handle it. That too would be nice to know, if it's the real driver of their actions.
Entirely rational behaviour (in the same position I'd do the same), but while an entertaining story, it's a hard to story to feel enthusiastic about even though in every other way it shows great leadership.
1. freeze registration of new accounts and purchase only the drives necessary to keep existing contractual obligations
2. charge new accounts at a different rate, promising to drop the rate once drive prices returned to normal
3. change plan storage volume limits
The only reason they didn't pick any of these 3 options, as far as I can tell, is because all of them were bad for their profit margins and it would be nice if they were upfront about that as the real driver for their actions. It's not clear that their business would have failed had they chosen one of these options.
So they chose a path that would screw over the little guy needing a new hard drive in San Francisco. Perhaps I'm wrong, though, and the timing made it very difficult to act on these other options, or their cashflow couldn't handle it. That too would be nice to know, if it's the real driver of their actions.
Entirely rational behaviour (in the same position I'd do the same), but while an entertaining story, it's a hard to story to feel enthusiastic about even though in every other way it shows great leadership.
[deleted]
The fact that external drives that have to be "shucked" of their casing and circuit boards (at what environmental expense?) were cheaper than the same internal drives makes you wonder whether the internal drive prices were realistic.
It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it.
Sometimes capitalism isn't perfect.
It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it.
Sometimes capitalism isn't perfect.
Partly because Costco charges only a 15% markup to the cost they acquired the goods at (http://en.wikipedia.org/wiki/Costco#Sales_model), which means that the external drives, purchased before the flood, were able to be sold at prices below the newly raised internal drive prices.
Were they the same as the internal drives? Or just the same capacity?
I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.
I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.
Gleb from Backblaze here. The drives were literally the same inside the external enclosures. We bought a few to start with to check.
We use consumer-grade drives across our entire 40+ petabyte storage farm because we do not need high-performance drives. Instead, we need low-cost, low-power, high-density drives to put into our custom-designed Storage Pods: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
We tested "enterprise drives" and found that from a pure reliability perspective, they actually performed worse in our testing than the consumer drives. With about 5 years and 40+ petabytes of data, we're fairly confident this holds true.
Finally, we assume that all drives will fail and replace about 10 drives every week. The software is written such that the system takes drive failures into account and data is stored redundantly and continuously cross-checked.
We use consumer-grade drives across our entire 40+ petabyte storage farm because we do not need high-performance drives. Instead, we need low-cost, low-power, high-density drives to put into our custom-designed Storage Pods: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
We tested "enterprise drives" and found that from a pure reliability perspective, they actually performed worse in our testing than the consumer drives. With about 5 years and 40+ petabytes of data, we're fairly confident this holds true.
Finally, we assume that all drives will fail and replace about 10 drives every week. The software is written such that the system takes drive failures into account and data is stored redundantly and continuously cross-checked.
It's interesting to find that "enterprise drives" performed worse than consumer. Recently, a report came from a Microsoft datacenter, where they found most errors weren't from faulty hardware, but firmware incompatibilities between array controller and drive controller. I wonder if this type of incompatibility occurs more often for enterprise drives than consumer.
The prices on external drives lagged the prices on internal. The externals are generally sold with reduced warranty -- 1 year, or sometimes as low as 90 days -- and my suspicion has always been that many of them failed an internal longevity test and were binned.
So Backblaze is facing a huge increase in drive failure rates in the near future? That's reassuring.
(Backblaze engineer here.) I have no magic crystal ball, but we do have more than 10,000 hard drives spinning in our datacenter, some for 5 years now. We have collected some pretty detailed statistics regarding failure rates vs. drive brand (and 20 other metrics). We really don't see the correlation you imply (shucked consumer drives failure rates). I'm hoping to do a blog post about what we have found regarding drive failure rates, but here is a TL;DR summary: drives fail approximately 1 - 5 % each year, your application and RAID architecture and monitoring strategy MUST compensate for that with no data loss.
I'd love to see more information from people with many drives about reliability.
The well known Google study was interesting.
The well known Google study was interesting.
Another way to look at it is they potentially found an inexpensive form of credit. They save money now and might have to repay it back later -- the interesting question will be how much "credit" they'll have to "repay" because of a higher failure rate.
I hope they're keeping some basic metadata about the shucked drives, so they can compile some statistics for us. It would be fascinating to see what the lifespan of these drives will be over the next few years!
I hope they're keeping some basic metadata about the shucked drives, so they can compile some statistics for us. It would be fascinating to see what the lifespan of these drives will be over the next few years!
That could be a lot of liability they're holding, since they voided the warranties on the externals by shucking them. So they could wind up paying twice for failed HDDs = $130 + $130 + $25 in gas and wage to pay someone to drive around to pick these up = $285 or so per HDD?
Plus 2 people got banned from Costco.
Plus 2 people got banned from Costco.
Failure rates are still likely to be low (<5% per annum)
I've only ever bought five external drives in my lifetime. Incidentally, all of them have been "shucked" over the years, and none of them have failed. They vary from 3-5 years old.
In fact, the only two drives I've had fail on me were OEM packaged, so based on my anecdotal, statistically insignificant personal experience you could even say the opposite, that external drives are more reliable than their OEM counterparts.
You never know. Perhaps the additional packaging prevents damage to the drives in transit, and the process of installing software on the drives (as I believe some manufacturers do with external drives) provides another opportunity to quality check the drives.
I've only ever bought five external drives in my lifetime. Incidentally, all of them have been "shucked" over the years, and none of them have failed. They vary from 3-5 years old.
In fact, the only two drives I've had fail on me were OEM packaged, so based on my anecdotal, statistically insignificant personal experience you could even say the opposite, that external drives are more reliable than their OEM counterparts.
You never know. Perhaps the additional packaging prevents damage to the drives in transit, and the process of installing software on the drives (as I believe some manufacturers do with external drives) provides another opportunity to quality check the drives.
*Yev from Backblaze here: I can now purchase from Costco again, so all is well on that front. Even with a price at or around your calculated $285, that is still half as expensive as the $600 per drive we were once quoted during the crisis.
They may have different branding or model numbers, but in terms of technical specifications they are no different to internal drives.
The article implies that all internal drives rocketed in price. The drives used in the external drives will most likely be a subset of the internal drives available.
(Backblaze engineer here.) Exactly, we chose the particular external drives to contain the exact type of drives that we prefer in our datacenter.
MyBook Essentials (pictured) use the same Caviar Green internal hard drive, with the same warranty, which is still priced $10 more on amazon.
Yes, think of the bare drive (if you don't need the enclosure) as "frustration free packaging" - ie, the kind that Amazon charges a good % markup - and people willingly pay for.
Sure it is. You are watching capitalism in action. The company that could effectively utilize shucked hard drives did so. Larger companies that don't have the management oversight to follow this type of detail just let them be fore sale at what they already were.
Most of the drives in external devices are "green" drives which has higher seek time and/or lower RPM. The reason why these drives' price have not increased as much/as fast is because the demand is lower than the "regular" internal drives.
I did the same thing when the "crisis" happened, I bought a load of external drive and removed the hard disk from them.
I did the same thing when the "crisis" happened, I bought a load of external drive and removed the hard disk from them.
What is a "green" drive?
It's a HDD which is supposed to use less electricity because it has a lower or variable spin speed. It's probably mostly a marketing gimmick.
It's not a gimmick, but it's also a double-edged sword. They also do fun things like spinning down on an aggressive timeout, which can lead to early SMART warnings about head load/unload cycles if you keep waking them up to write periodic log files to them. I had to "fix" this with a nasty little DOS utility on a home Linux box I built using Seagate's version of this.
... and you should avoid them at all costs for any serious application. To this day there is still confusion between host adaptors (raid controllers) and green drives, and the firmware management process (you will absolutely need to manage the firmware on these) is atrocious.
The "green" is certainly a gimmick. But the lower power usage is not. 10 Backblaze Storage Pods full of "green" drives use the same amount of power as 8 pods full of "regular" drives.
Can you spin all the drives down in a pod once the pod is full of data until a restore is needed?
Don't modern OSes send spin-down commands to HDDs when not in use?
Exactly what stephengillie said, here's an example http://www.wdc.com/en/products/internal/desktop/
Capitalism doesn't guarantee that things are priced in order of production costs. In particular, after these goods were produced, the ongoing inventory situation for hard drives had a wrench thrown into the works. One would hope the market then allocated a once-commodity component efficiently, such that users with the highest need for hard drives received them and users with less need made the decision to purchase hard drives later. For social (non-market) reasons, some people who sold hard drives decided to essentially sell hard drives below their market value, with rationing, to a) maintain availability of supply for a market segment with a relatively low need for hard drives while b) avoiding selling those hard drives into a market segment with a relatively high need for hard drives.
Economics (not quite capitalism) says that if you attempt to replace a market with a rationing system, you will get a market anyhow, because markets are emergent features of reality and as soon as you have multiple agents with differing values for the same goods trade happens. (Capitalism would add "... and that's a good thing!") Rationing will generally result in there being an artificial consumer surplus created in the system versus a functioning market, and that consumer surplus will generally be captured by a market-maker rather than either any of the intended beneficiaries of the rationing scheme or the entity who has the product requiring rationing in the first place.
In the absence of a rationing scheme, a) Backblaze would have had as many drives as it wished to purchase at the market price (slightly lower than the price they think is unconscionable in their normal channels) and b) most home users of hard drives would have been unable to source new hard drives at rates they were willing to pay, and would have turned to substitutes like "use refurbished hard drives", "decrease consumption of hard drive space", and "defer hard drive purchases."
Economics (not quite capitalism) says that if you attempt to replace a market with a rationing system, you will get a market anyhow, because markets are emergent features of reality and as soon as you have multiple agents with differing values for the same goods trade happens. (Capitalism would add "... and that's a good thing!") Rationing will generally result in there being an artificial consumer surplus created in the system versus a functioning market, and that consumer surplus will generally be captured by a market-maker rather than either any of the intended beneficiaries of the rationing scheme or the entity who has the product requiring rationing in the first place.
In the absence of a rationing scheme, a) Backblaze would have had as many drives as it wished to purchase at the market price (slightly lower than the price they think is unconscionable in their normal channels) and b) most home users of hard drives would have been unable to source new hard drives at rates they were willing to pay, and would have turned to substitutes like "use refurbished hard drives", "decrease consumption of hard drive space", and "defer hard drive purchases."
Those are loss leaders. Much like the 29c/lb turkey at Thanksgiving, or the $29 laptops the day after, these deals are designed to get consumers into the store or website, so each will buy 1 or 2 and several other items and accessories. High-margin accessories like USB cables and wireless mice, or potatoes and stuffing, easily offset those loss leaders.
The entire point of the 2 item limit on deals like those is to prevent companies from doing exactly what Backblaze has done. By snapping up all of these deals, they're reducing the number available for consumer purchase, which impacts the revenues of these big-box retailers. Basically, Backblaze absorbed the (economic) consumer and retailer surpluses for HDDs in the Bay area and passed it on to their consumers.
The entire point of the 2 item limit on deals like those is to prevent companies from doing exactly what Backblaze has done. By snapping up all of these deals, they're reducing the number available for consumer purchase, which impacts the revenues of these big-box retailers. Basically, Backblaze absorbed the (economic) consumer and retailer surpluses for HDDs in the Bay area and passed it on to their consumers.
I agree with the gist of this, but it's worth noting that the drives that you typically buy for arrays like this are enterprise SATA drives (or at least 24/7 duty cycle desktop drives) and the drives inside the enclosures are the bottom of the barrel ...
(Backblaze engineer here.) Online backup is a unique application for hard drives. This is not a high transaction environment like a database getting pummeled with SQL queries. In online backup, we write the encrypted data only once, remember a SHA-1 to verify the data later, then we read it once a week to make sure the data still has perfect integrity (the SHA-1 must match). We now have over 10,000 hard drives spinning in our datacenter, some for over five years, and we've collected some pretty detailed statistics on drive failure rates vs many parameters. For our application (online backup) drives from external enclosures and consumer drives last every bit as long in our environment as "enterprise" drives.
I'm sure you guys keep track of read/write statistics as well; have you thought about spinning drives down on full BB pods until a restore is requested (similar to how Amazon Glacier's storage system works)?
I read once that the majority of drive failure happens during transitions to or from a running state, and thus it is counter-intuitively safer to just leave them running all the time.
I am sure there is some mathematical formula that will express the risk based on how infrequently one expects to spin the drive up (and then down again), but I don't know it.
I am sure there is some mathematical formula that will express the risk based on how infrequently one expects to spin the drive up (and then down again), but I don't know it.
Backblaze is hosted out of a SF datacenter, where power is much more expensive than other US locations; I assume someone has done the math whether its cheaper to replace failed drives from frequent spinups/downs vs burning the power to keep them spinning.
We had this problem at rsync.net ... the price of our main consumable tripling overnight, etc.
We really never considered "shucking" external drives, since the drives inside these enclosures are the worst spec'd, highest failure rate drives ... never mind the (usually) nonsensical enterprise vs. non-enterprise, they're just crummy drives.
Good for BB for making it work ...
BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise drives two months ago were $198 (or so) and last week they were $248 (or so).
We really never considered "shucking" external drives, since the drives inside these enclosures are the worst spec'd, highest failure rate drives ... never mind the (usually) nonsensical enterprise vs. non-enterprise, they're just crummy drives.
Good for BB for making it work ...
BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise drives two months ago were $198 (or so) and last week they were $248 (or so).
But Blackblaze's infrastructure is properly designed to tolerate hw failures. They use RAID6 and replicate your data to 2 or 3 different storage pods. Smarter than beint forced to buy "better" expensive drives IMHO when your primary business expense is raw storage.
If a disk fails, it still needs to be replaced. RAID etc lets you avoid data loss until you replace the drive.
If BackBlaze are adding 50TB of data a day, are they really buying drives from NewEgg?
That's one of the things that caught my attention as well. It seems strange, I would've imagined that at 50TB/day (~500 x 3TB HDDs/month) they could get a nice deal with a direct importer. I guess there is a small PR twist to the whole story ;-) with newegg being a well-known name.
I don't know if the 50TB/day figure includes the replacement drives or only expanding the capacity as I guess their failure rate is quite high.
edit: fixed a typo.
I don't know if the 50TB/day figure includes the replacement drives or only expanding the capacity as I guess their failure rate is quite high.
edit: fixed a typo.
My guess would be that the 50TB/day figure is before de-duplication and not at all relevant to the actual storage, but even at 10% of that I would expect them to be sourcing them wholesale, not retail.
In fact, this whole article is making me doubt my faith in Backblaze as an off-site backup solution.
In fact, this whole article is making me doubt my faith in Backblaze as an off-site backup solution.
I remember a reddit AMA where they mentioned that they don't actually do any de-duplication. This isn't Dropbox and nowhere as reliable as S3 - its literally as if you're renting remote consumer external drives for offsite backup, for that usage pattern Backblaze makes some sense.
And those pods are cool. But I agree, a somewhat flawed approach.
And those pods are cool. But I agree, a somewhat flawed approach.
>This isn't Dropbox and nowhere as reliable as S3 - its literally as if you're renting remote consumer external drives for offsite backup
What are their reliability promises? I think that they're doing some mirroring of the data, but I'm still not sure about the rest of their service details.
I'm considering using them for backups, but have not decided yet.
I came upon this [1] discussion here, from 3 years ago. I'm diving in :)
Yeah, the pods are really sweet and they provide all the design documents [2]!
[1]: http://news.ycombinator.com/item?id=803136
[2]: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
edit: added additional details about the design
What are their reliability promises? I think that they're doing some mirroring of the data, but I'm still not sure about the rest of their service details.
I'm considering using them for backups, but have not decided yet.
I came upon this [1] discussion here, from 3 years ago. I'm diving in :)
Yeah, the pods are really sweet and they provide all the design documents [2]!
[1]: http://news.ycombinator.com/item?id=803136
[2]: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
edit: added additional details about the design
I don't think it should be assumed that there is no data redundancy between pods, afterall from prior blog posts it seems like the pods can be taken offline. I'd also posit that a properly constructed RAID array can mitigate the risks of using consumer parts; afterall that is the "I" in RAID.
[deleted]
Why not let NewEgg hold the inventory if their prices are good?
NewEgg isn't just retail anymore. They have a 'NewEgg Business' program with pretty solid volume discounts.
A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I could order them individually from Amazon for £27 each with free delivery.
I phoned up all the official Microsoft distributors in my country, who phoned up Microsoft and whatnot. Not one of them could offer me a price below about £35. And getting in contact with Microsoft hardware directly? Ha, I should be so lucky.
I phoned up all the official Microsoft distributors in my country, who phoned up Microsoft and whatnot. Not one of them could offer me a price below about £35. And getting in contact with Microsoft hardware directly? Ha, I should be so lucky.
A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams.
I'll bite. Why? :)
I'll bite. Why? :)
Asked to quote to make a system like [1] for a fleet of 1000 vans. Buying in USB cameras would reduce engineering costs compared to making our own. Eventually went for a brand that cost less and could accept wide angle lenses.
[1] http://www.smartdrive.net
[1] http://www.smartdrive.net
More often than not it's a case of who you know when it comes to stuff like this.
(Backblaze engineer here.) We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet, thus we NORMALLY bid out each drive purchase (about one large order per month) to four or five regular "resellers" and "distributors" like CDW, Ingram, B&H, etc and take the best price that month. We were down at $90 or so for individual Hitachi 3 TBytes at the low point before the crisis. We're currently paying around $160 from CDW and friends and we can afford that without raising our prices. During the Thailand flood crisis, CDW offered "not enough drives" (half our order) for $350 per drive. We said "no", because we couldn't afford "yes". :-)
Anybody can win our business by under bidding the other resellers, and we're ALWAYS open to suggestions. If you know any way to achieve lower drive prices, let us know! PLEASE!!
Anybody can win our business by under bidding the other resellers, and we're ALWAYS open to suggestions. If you know any way to achieve lower drive prices, let us know! PLEASE!!
I don't understand this comment at all. If anything, it has redoubled my interest in remaining a (paying) customer of BackBlaze. Look at the lengths they will go to securing drives so that I can continue to have stupid-cheap backup!
I can't ask for more.
And realistically, if push comes to shove, I'm sure BackBlaze would send out a heartfelt email to their loyal customers explaining the situation and kindly requesting a price increase. Especially after reading a post like the one they just submitted...I'm sure no one would have a problem ponying up a few more dollars.
I can't ask for more.
And realistically, if push comes to shove, I'm sure BackBlaze would send out a heartfelt email to their loyal customers explaining the situation and kindly requesting a price increase. Especially after reading a post like the one they just submitted...I'm sure no one would have a problem ponying up a few more dollars.
If I were their customer, I would be looking for alternatives right about now...
Why would you do that? Due to reliability concerns, since they're using 3TB SATA drives, sourced by Cotsco and Best Buy?
I'd be shocked if they were not the same drives you buy from a supplier.
I mostly agree with this. That's why I asked about GP's motivation.
I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives).
If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time.
So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable).
edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
I suspect, though, without any actual data to back this up, that the drives that go into (low-cost) external HDD enclosures have lower MTBF ratings than the desktop-oriented internal drives. I think that the usage pattern for an external HDD is very different in terms of longevity in operation than desktop hard drives (and these have lower guarantees than eg. server-grade SATA drives).
If I was a manufacturer of such enclosures I'd be willing to buy cheaper drives with lower promised MTBF for embedding them in enclosures that stay off most of the time.
So, even if they're the same, I guess the manufacturer performs some tests and sells drives that fail some non-critical ones at a lower price (pretty much what Intel and AMD do with their chips - those that can't sustain the highest frequencies are downgraded to clock rates that are stable).
edit: The comment from @dsr_ tends to support my hypothesis, although I haven't looked at the differences in guarantees of external enclosures, it's true that internal desktop HDDs can have up to 5 years of guarantee (i.e. WD's Black series has a 5 years guarantee).
But then Backblaze's use of HDDs is not the same as an HDD stuck in a desktop machine and used as a primary/secondary drive.
It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours).
AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod.
Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.)
(Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.)
If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased.
HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.
It's why their restore isn't instantaneous. You have to ask for the files and they'll let you know when they're ready, this takes time (up to several hours).
AIUI they essentially fill up each Pod with incoming data and, when it's full it goes into a low power mode where they spin down all of the disks in that Pod.
Restore requests (from many users) will usually span quite a few Pods so they batch up requests for each Pod so that can minimise the number of times that drives in a Pod need to be spun up to copy off files. Rather than spinning up and down once every 10 minutes to service 36 requests over 6 hours they wait 6 hours, spin up once and service all 36 requests for that pod in one go. (Figures are for example only.)
(Small restores may be instantaneous if the Pod is still being filled or just happens to be spun up for restore soon. Large restores will more than likely span multiple pods and so there's more of a delay until all parts of the restore have been obtained from all of the pods involved.)
If anything this maps pretty well to the expected usage pattern of an HDD in an external enclosure as many of them are used for infrequent backups rather than extending constant use storage. It's how I expect Backblaze will be hoping to get much more than the usual MTBF out of their drives they have purchased.
HDD guarantees may be time based, but HDD failure rates are linked much more closely with usage patterns; number of spin-ups/downs, number of seeks, number of sector writes, etc. A drive in constant use on a desktop machine is going to be hammered a lot more than one that's essentially written to just once (as the pod fills up) and then spun up every now and again for partial reads.
If thats the access pattern, couldn't they just use tape? Of course you can't easily bootstrap an automated warehouse for these things, but once you have it, you enjoy the drastically lower prices and better availability of tape.
(I know nothing about using tapes to seriously store data, so this is conjecture.)
(I know nothing about using tapes to seriously store data, so this is conjecture.)
Having a tape libarary/silo is much harder to scale vs building pods of hard drives.
Experience: I helped route and process data for the Large Hadron's CMS detector.
Experience: I helped route and process data for the Large Hadron's CMS detector.
Yev from Backblaze here -> The drives were EXACTLY the same as the ones we would put in to our storage pods prior to the crisis. We selected specific external drives to break open and check because we thought they might be similar, and we were right. The drives ended up being the same, just put in to an external shell. We've always used consumer level products to avoid the "enterprise tax", so reliability more or less stayed constant.
The post struck me as slightly unprofessional, but from a technical standpoint I don't see anything that would change my opinion of the service they provide.
They achieve the best bang for buck by using ordinary consumer hard drives in RAID arrays. During the hard drive crisis, the cheapest way of procuring these was by purchasing external drives from ordinary stores.
There's nothing wrong with that. If anything, it shows agility and ingenuity.
They achieve the best bang for buck by using ordinary consumer hard drives in RAID arrays. During the hard drive crisis, the cheapest way of procuring these was by purchasing external drives from ordinary stores.
There's nothing wrong with that. If anything, it shows agility and ingenuity.
And, I would add, intense vulnerability to future market fluctuations. If this was their contingency plan, they've exhausted it. What happens next time?
This was a one-off event that I'm sure the hard drive manufacturers themselves will be taking steps to prevent reoccurring.
What else can Backblaze do? Stockpile drives? The hard drive industry moves fast, with major product releases every 6-12 months and constant price reductions; this would almost certainly increase the cost to the end user.
No business can adequately prepare for a supply shock like this. Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business.
In this case, would you be berating the ingenious airline that goes out of its way to track down fuel at a cheaper price, finding a low-cost fuel that they can easily refine themselves to aviation fuel, so as to honour their commitments to their customers and avoiding raising their prices?
What else can Backblaze do? Stockpile drives? The hard drive industry moves fast, with major product releases every 6-12 months and constant price reductions; this would almost certainly increase the cost to the end user.
No business can adequately prepare for a supply shock like this. Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business.
In this case, would you be berating the ingenious airline that goes out of its way to track down fuel at a cheaper price, finding a low-cost fuel that they can easily refine themselves to aviation fuel, so as to honour their commitments to their customers and avoiding raising their prices?
> Imagine airlines facing a 300% increase in the price of fuel. Almost every single one would go out of business.
No. They use hedges to control their cost (this is the original point of hedging...). If a company is reliant on a constant supply of hard-drives and doesn't want to be exposed to the risk of extreme price jumps it should probably do the same.
No. They use hedges to control their cost (this is the original point of hedging...). If a company is reliant on a constant supply of hard-drives and doesn't want to be exposed to the risk of extreme price jumps it should probably do the same.
Airlines have long term fuel contracts to smooth out the short term price changes, but beyond that they just pass it on to consumers.
e.g. http://www.delta.com/business_programs_services/delta_cargo/... the first column is the fuel surcharge that floats with fuel prices.
The Backblaze article clearly said that paying more for drives and passing it on the customers was an option. They just found a better one. They didn't mention it, but just losing money for a while is also a reasonable option if you don't want to antagonize your customers.
e.g. http://www.delta.com/business_programs_services/delta_cargo/... the first column is the fuel surcharge that floats with fuel prices.
The Backblaze article clearly said that paying more for drives and passing it on the customers was an option. They just found a better one. They didn't mention it, but just losing money for a while is also a reasonable option if you don't want to antagonize your customers.
*Yev from Backblaze here -> You're correct, but quick note on this. We're a boostrapped startup, meaning, we need to be constantly profitable in order to pay salary and keep the business going, taking a financial hit was the only real option we could not take, because we would have been out of business very quickly (not good for anyone).
[deleted]
Hedges hit you both ways, though. They save you if prices spike, but also cost you if prices drop.
Given that hard drive prices almost always drop constantly, is that tactic really a good idea? Is it possible to reasonably hedge against price spikes in a commodity whose price you expect to consistently and rapidly drop? (Serious questions, really.)
Given that hard drive prices almost always drop constantly, is that tactic really a good idea? Is it possible to reasonably hedge against price spikes in a commodity whose price you expect to consistently and rapidly drop? (Serious questions, really.)
If prices are actually expected to drop, that should already be factored into the price of hedging the commodity.
How would that work, exactly? Would you set up a contract whereby you can buy hard drives according to a price schedule which decreases according to a set schedule, where that price decrease has you paying a bit more than the expected market price, but is guaranteed and thus protects you from increases?
If prices are expected to drop, your hedge would be for that somewhere around the price you expect it to be wouldn't it? Say you expect 3TB hard drives to be $110 in 3 months and the current price is $130. You may buy a contract for $115/hd - if someone is willing to sell you that contract.
Typically if you're hedging a commodity for actual use, what you're trying to do is to lock in your cost and not to make money out of it. Let's look at an airline for example. The price of oil currently is $90/barrel and I think it might go up even more to $120/barrel. I buy a contract for 100 barrels of oil at $100/barrel deliverable in 3 months. Say I use 100 barrels of oil every 3 months. I now know that my fuel cost is now $10,000 for those 3 months. This then allows me to price my product based on that $10,000 assumption.
In the case of hard drives, I have no idea if it would be a good idea to hedge since the prices keep on dropping and the Thailand flood is not an everyday occurance and in general prices of hard drives are pretty stable and don't fluctuate like oil prices. But fundamentally, you're not trying to avoid losing money; you're trying to lock in your cost so you can price your service accordingly and not be in a bind if the price of your commodity fluctuates.
Typically if you're hedging a commodity for actual use, what you're trying to do is to lock in your cost and not to make money out of it. Let's look at an airline for example. The price of oil currently is $90/barrel and I think it might go up even more to $120/barrel. I buy a contract for 100 barrels of oil at $100/barrel deliverable in 3 months. Say I use 100 barrels of oil every 3 months. I now know that my fuel cost is now $10,000 for those 3 months. This then allows me to price my product based on that $10,000 assumption.
In the case of hard drives, I have no idea if it would be a good idea to hedge since the prices keep on dropping and the Thailand flood is not an everyday occurance and in general prices of hard drives are pretty stable and don't fluctuate like oil prices. But fundamentally, you're not trying to avoid losing money; you're trying to lock in your cost so you can price your service accordingly and not be in a bind if the price of your commodity fluctuates.
In what real-world scenario would airlines respond to a short-term rise in the cost of fuel by finding a new low-cost fuel and refining it themselves to fuel engines designed and tested with existing fuels?
Obviously I'm no expert on aviation fuels, I'm just trying to make an equivalent (most probably fictional) scenario for the industry.
In this case internal drives are ordinary aviation fuel, external drives are the alternative fuel, the price of which has been largely unaffected by the supply shock, and the process of taking the external drives apart to reveal internal drives is the equivalent of a simple refining process that turns the alternative fuel into ordinary aviation fuel.
In this case internal drives are ordinary aviation fuel, external drives are the alternative fuel, the price of which has been largely unaffected by the supply shock, and the process of taking the external drives apart to reveal internal drives is the equivalent of a simple refining process that turns the alternative fuel into ordinary aviation fuel.
Did you miss the part where they had a month's supply and the option of just paying more on the OEM market? This is just how they made best use of the market by going to nontraditional channels, not their only option.
This sort of thing happens all the time in business. There are a variety of common solutions, for example borrowing and repaying once prices are back to normal again, throttling demand in other ways, raising prices to the customer, cutting costs in other parts of the business, etc. All of those affect the value to the customer in some way. They managed to find a creative way to limit that effect.
In the end, they are cheap, if you are worried about their methods I'm sure they have competitors who approach things in a more traditional fashion and charge accordingly.
In the end, they are cheap, if you are worried about their methods I'm sure they have competitors who approach things in a more traditional fashion and charge accordingly.
Why?
They have built a backup architecture designed to use low reliability commodity hard drives.
This post shows the lengths they go to just to keep growing. Imagine what they'll do to keep the system running!
They have built a backup architecture designed to use low reliability commodity hard drives.
This post shows the lengths they go to just to keep growing. Imagine what they'll do to keep the system running!
I am a customer of theirs both of their backup service and their plans for machines. The backup service has never let me down and at work we've built two of their machines using their plans saving over 10K.
The machine plans are on their site and they explain why they chose off the shelf components. My only regret is that no one on our team has welding skills, so we couldn't make awesome face plates. lol
Backblaze is one of the more transparent companies operating in the space, which increased the trust level.
The machine plans are on their site and they explain why they chose off the shelf components. My only regret is that no one on our team has welding skills, so we couldn't make awesome face plates. lol
Backblaze is one of the more transparent companies operating in the space, which increased the trust level.
Gleb from Backblaze here. Glad you're enjoying the service and the spec design. As for the faceplates, our welding skills aren't great either. However, the company we had cut our steel case for us (www.protocase.com) will cut a custom logo for you into your faceplate as well.
Although I am in the process of moving away from Backblaze, it's not because of this. Rather it's because they don't provide the flexibility I've come to need for prioritizing content to backup, and when I recently ran into a problem after swapping out hard drives, the error message Backblaze provided was the most unhelpful piece of drivel I've ever seen. --I was told to contact support with the error code "horse" rather than just being told that it was a timeout issue!
*Yev from Backblaze here -> Yea, that code was very early and was intended for support to gather data from the field as we were collecting data on what was causing the timeout, the message was to prompt folks to write in so they could collect some additional information. We'll probably be changing it in the near future!
I am a Backblaze customer and this post made me admire their commitment to provide their service even in times of crisis. I won't be switching away, and I don't see why I would.
same news ; and comments:
http://news.ycombinator.com/item?id=4630910
http://news.ycombinator.com/item?id=4630910
This one is the original article, and is better-written too.
(Gigaom blogspammed it.)
(Gigaom blogspammed it.)
I wonder if the hard drive manufacturers will reconsider locating their entire supply chain in a third world country with frequent natural disasters after this incident. With rising wages in the third world and stagnating wages in America, perhaps the financial benefits of outsourcing aren't so apparent anymore.
They have more data about all the pieces and processes leading to their ultimate market share and profitability so I have no doubt they try to maximize those.
I suspect the biggest issues are that all their suppliers and customers are in that region. Think fabs to make the chips, the various motors and related electronics, coating processes, and then sending them off to the places that make HP/Dell/etc systems. If just the drive factory were moved then you'd still have the drive parts crossing the ocean to get to the US and the completed drives crossing once again to go back into the systems.
Usually the relocations happen when a disruptive solution comes along. You get the advantage of a somewhat clean sheet. For example SSDs don't need all the mechanical components and fine tuning, so they can be made with a completely different supply chain.
It won't happen in hard drives but for other products customization is what can spur relocation. For example Lenovo is opening a manufacturing plant in the US to deal with customization which allows quicker response times. (Customized orders often can charge a little more which offsets increased costs.)
I suspect the biggest issues are that all their suppliers and customers are in that region. Think fabs to make the chips, the various motors and related electronics, coating processes, and then sending them off to the places that make HP/Dell/etc systems. If just the drive factory were moved then you'd still have the drive parts crossing the ocean to get to the US and the completed drives crossing once again to go back into the systems.
Usually the relocations happen when a disruptive solution comes along. You get the advantage of a somewhat clean sheet. For example SSDs don't need all the mechanical components and fine tuning, so they can be made with a completely different supply chain.
It won't happen in hard drives but for other products customization is what can spur relocation. For example Lenovo is opening a manufacturing plant in the US to deal with customization which allows quicker response times. (Customized orders often can charge a little more which offsets increased costs.)
Now that's a startup story. Backblaze has some of the best down home old fashioned spit n glue startup stories on hacker news. Like the one where they invented their own storage pod out of commodity components to beat amazon's pricing. And now this stay-alive-against-all-odds story. Fantastic.
I was impressed with the effort to get those drives. If the massive companies who produce hard drives were unprepared for such a natural disaster I can't say I expect more from a small company.
[deleted]
Their execution was poor. Start by depleting SF and then look elsewhere? Short-sighted at best.
They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over.
I give them a C- on execution.
They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over.
I give them a C- on execution.
Yeah sure. In a weekend they could have purchased 1500 hard drives when there is a shortage and stores are limiting them to 2? I would challenge you to go to retail stores in a single day under those conditions and buy 25.
That's predicated on them being able to spare many employees just for drive sourcing, which as a relatively small company they probably couldn't. They went with the lower cost stop gap at first, if the shortage hadn't lasted as long they would have been better off with how they did it.
the hard drive shortage hit us all. this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use.
while i don't think this is illegal, it doesn't make me feel warm and fuzzy inside either.
when there's a shortage, the "noise" in traditional distribution means that small buyers (like me, and many others) can still find what we need if we put in some extra effort. it functions like a "soft buffer", making the transition to a complete lack of resources more gradual. action like this removes that buffer.
this feels like one group of people - driven by their desire to do business - beat out a load of other people who didn't have the same logistical resources. there's something uncool about it. i am not sure i have explained very well, but it's to do with crossing the line from business to personal.
[edit: ah, thanks - separate thread: http://news.ycombinator.com/item?id=4630910]
[edit2: wtf? i thought someone replied here? was about to upvote you. thanks anyway.]